glob expansion should ignore files that cannot be stat()'ed

[Kurtis Rader]

@hanche noticed that stat /var/db/DifferentialPrivacy fails with error "operation not permitted". This is because of a macOS security mechanism known as SIP. This causes Elvish glob expansion to short-circuit when it is unable to stat() the file:

~> cd /var/db
> put *
▶ Accessibility
▶ BootCache.data
▶ BootCache.playlist
▶ BootCaches
▶ CVMS
▶ ConfigurationProfiles
▶ CoreDuet
▶ DetachedSignatures
▶ DiagnosticPipeline

This is related to issue #1240 that I fixed almost a year ago. It didn't occur to me at the time that there could be scenarios, short of a broken filesystem, that stat() could fail. The simplest solution is to silently ignore any file that cannot be stat()'ed. However, in the context of a simple glob expansion we should still be able to include the problematic path name in the generated output.

[Kurtis Rader]

I've got a tentative fix for the glob expansion problem involving files that are protected by macOS SIP. With my change I can execute put /var/db/* and enumerate every file in that directory; including /var/db/DifferentialPrivacy. It even passes all the existing unit tests. However, this is a pretty big change to the behavior of glob expansion so I want to cogitate on my fix for a few days before opening a pull-request. In particular, it's not at all obvious how to verify the change since this is a corner case that is hard to simulate with the current Elvish unit test infrastructure.

With my fix this now works:

~> put /var/db/*
▶ /var/db/Accessibility
▶ /var/db/BootCache.data
▶ /var/db/BootCaches
▶ /var/db/CVMS
▶ /var/db/ConfigurationProfiles
▶ /var/db/CoreDuet
▶ /var/db/DiagnosticPipeline
▶ /var/db/DiagnosticsReporter
▶ /var/db/DifferentialPrivacy
▶ /var/db/DuetActivityScheduler
...
▶ /var/db/vmware
▶ /var/db/volinfo.database

[Kurtis Rader]

I'll also note that by "pretty big change" I do not mean the number of lines changed by my fix. That is actually quite small. It's a big change in the sense that glob expansion now ignores EPERM errors (on Unix) from os.Lstat() and it's unclear what the equivalent is on Windows.

[Kurtis Rader]

On Windows fs.ErrPermission corresponds to ERROR_ACCESS_DENIED (defined in https://pkg.go.dev/golang.org/x/sys/windows). This does seem to be directly analogous to EPERM on Unix platforms and is probably safe to use in the same manner on Windows. It is still unclear whether there are other corner cases involving that error value but I am satisfied that special-casing it in the Elvish glob expansion code is justified and safe to do.

TBD is how to augment the existing glob unit tests to verify the new handling of a "permission denied" error returned by os.Lstat(), or even if that complication is justified.

[Kurtis Rader]

I'm glad I slept on my solution and tried to find ways to break it. I expected os.Lstat() to return a "zero value" fs.FileInfo structure if an error (such as EPERM) occurred. However, it instead returns a nil pointer because fs.FileInfo is an interface, not a structure. Which means that passing the value, when an error is reported, to the callback can panic when it receives an unexpected nil pointer. Such as when doing put /var/db/*[type:regular]. My expectation was that a "zero valued" fs.FileInfo structure would simply fail most checks (such as whether the path was a regular file or directory) if the os.Lstat() call returned an error and therefore was safe to use. /facepalm

So fixing this is going to be slightly more complicated than I had expected.