ValueError: underlying buffer has been detached (on Travis CI)
wookayin opened this issue · comments
On some environments (Travis CI on windows instances), Terminal() fails inside the constructor:
.eggs\blessed-1.16.1-py3.7.egg\blessed\terminal.py:185: in __init__
callable(stream.fileno) else None)
E ValueError: underlying buffer has been detached
In such an environment where stdout/filestream can be detached, sys.__stdout__ can be invalidated where sys.stdout does not. For a reference, please have a look at this: numpy/numpy#2761 (comment)
Suggestion: Why not sys.stdout?
Anyway Terminal() fails in some test environments (but works okay in normal use cases). It seems that the Travis-CI (windows) tests of blessed do not have this issue -- they seem to be using TestTerminal which is basically Terminal(kind=...) but they appear to be subjective to this problem.
Suggestion: Why not sys.stdout?
Because we have to deal directly with the console device and sys.stdout is not always attached to the console.
Are you using numpy? It looks like they introduced a bug here. Their goal was to force to UTF-8, but they didn't do it in a very good way as far as I can tell. Later they added a check so they didn't force it if it was already UTF-8. If it's not, they redefine sys.stdoutand detach sys.__stdout__ with this:
sys.stdout = io.TextIOWrapper(
sys.stdout.detach(), 'utf-8', errors, newline, line_buffering)
In 3.7 and later they should probably be using sys.stdout.reconfigure() and, in earlier versions on Python 3, possibly something like this:
sys.stdout = io.TextIOWrapper(
sys.stdout.buffer, 'utf-8', errors, newline, line_buffering)
I'm not sure if that has side effects of doing that, but the way they are doing it, they're breaking documented interfaces.
So I would say there isn't really much reason to check if sys.__stdout__ is detached because it shouldn't be unless you're doing things you probably shouldn't be doing. We could make it fail in a nicer way, but I feel like it should be failing in this case.
I suggest you file a bug with numpy to not break sys.__stdout__. For your tests, as a workaround, you can specify sys.stdout as your stream when instantiating Terminal() or force your encoding to 'UTF-8' before calling Terminal().
Thanks for the feedback, windows support was added this last release, and I'm adding Travis CI automatic tests on windows for blessed this next release, so its perfect timing.
I don't know much more to say, than that automatic tests on Windows for Terminals are certainly more difficult!
And as much as I love numpy, and utf8 (only, everywhere), I think @avylove is right here, this is a kind of "monkey patching" of sys.stdout is too wild to workaround on our side
If you want to suggest this code change,
try:
stream_fd = (stream.fileno() if hasattr(stream, 'fileno') and
callable(stream.fileno) else None)
- except io.UnsupportedOperation:
+ except io.UnsupportedOperation, ValueError:
stream_fd = None
that's fine with me, does that work?
@avylove Thanks for your detailed comments and explanation. I understand there must be some reason to use raw std.__sysout__, as described. In my project I don't think we are using numpy, but this can also happen due to other libraries or side-effects.
I think the change suggested by @jquast will work for me and it looks good to me (it'd be also great to put some comment/context about ValueError on the except statement for future references).
Sounds great, and very best wishes to you in Ann Arbor, I did live there a number of years ago :)
it'll be in next release, might take a few days we're trying to do a lot of cleanup
Awesome, thanks!
Actually, there are some more lines that are accessing to sys.__stdout__.fileno (I still get the same ValueError):
https://github.com/jquast/blessed/blob/6046de9/blessed/terminal.py#L193
self._init_descriptor = (sys.__stdout__.fileno() if stream_fd is None
else stream_fd)This might not be a comprehensive list as I cannot reproduce nor test it systemically at the very moment. There might be a bit more, but not sure.
@jquast please don't forget to include this follow-up in the 1.17.0 release!
Sorry to miss it today, it's late though, but we'll make a 1.17.1 minor re-release tomorrow
@wookayin I looked at your tests. It's strange they're failing at import. Partly that's because term=Terminal() is an argument to a method. I would advise you to can Terminal once and save it as a class instance variable, since setting up the Terminal does take some cycles, especially on Windows. Regardless, since it's failing so early, it seem like it could be an environment issue. That also seems strange to me because we're also using pytest on Travis and not having this issue. We are using a newer version and the tox setup is a bit different, so that could have something to do with it.
I'd like to get to the bottom of why it's failing, but it might be hard to figure that out in a CI environment.
As far as continuing to chase the problem with workarounds, maybe @jquast has some ideas, but I think it would be easier for you to just use term=Terminal(stream=sys.stdout) in your code.
@avylove Thanks. Actually I figured out how to reproduce in a standard linux system (with tty):
import sys
sys.stdout.detach()
sys.stdout = sys.stderr # to make standard REPL, print, etc. work
sys.__stdout__.fileno()
# ValueError: underlying buffer has been detached
import blessed; blessed.Terminal()
# ValueError: underlying buffer has been detachedTherefore, Terminal(stream=sys.stdout) would work but the Terminal instance should be created later on rather than at import (as you discussed). The blessed library itself should work even if stdout is detached.
In line 193
self._init_descriptor = (sys.__stdout__.fileno() if ...if we simply replaced with sys.stdout it would work without an error; I verified this is enough to make blessed.Terminal() work in the above scenario, but I would like to leave to your side about what is the best way to fix this. (I am not sure which one works better, between setting as None or just using sys.stdout or stream)
From my side, I can use the Terminal(stream=sys.stdout) workaround. Since this problem happens in some rare environments (like Travis CI) only, I can move forward without waiting for the new release; this workaround works well with the previous versions. Thanks for the great suggestion!
Actually even with Terminal(stream=sys.stdout) this problem still happens. The aforementioned line (L193: self._init_descriptor = (sys.__stdout__.fileno() ...) fails regardless of stream was explicitly given or not. So I will have to wait for the bugfix in order to make Travis test on Windows fully operable.
The example makes sense, but it's something that should never happen. Essentially, there's a bug somewhere in your stack. That said, it is happening, so failing gracefully creates a better experience, although, outside of testing, someone could hit this and wonder why they aren't getting any styling with no indication of why.
I forgot you're running under a CI environment. It makes sense it's still falling there because stdout is not attached to a console device. We can try setting self._init_descriptor to None with a similar try block. @jquast , is it ok to set self._kind to 'dumb' in this case? Otherwise Jinxed will need to be patched to allow None to be passed to get_term() and it will set it to 'unknown' if TERM is not set.
yes, 'dumb' is fine, thanks and good luck
I think this is covered in the pull request
Thanks for your feedback. I agree fixes like #136 is a good approach.
Now that the PR #136 has been merged, however, I think this issue still presents. The exception thrown from sys.__stdout__.fileno is ValueError, which is not captured by:
https://github.com/jquast/blessed/blob/0f8cd27/blessed/terminal.py#L212-L215
Just pushed a fix. Took out io.UnsupportedOperation because it's a subclass of ValueError.
Now it works well. Thank you!
We will push a minor release fix of these things to pypi by end of weekend if that helps you any, thanks for helping us @wookayin