[Bug] Reading menu bar events causes program to crash when displaying images with opencv
hatfield-c opened this issue · comments
Type of Issue (Enhancement, Error, Bug, Question)
Bug
Operating System
Windows 10 64-bit, Version 22H2
PySimpleGUI Port (tkinter, Qt, Wx, Web)
tkinter
Versions
Version information can be obtained by calling sg.main_get_debug_data()
Or you can print each version shown in ()
Python version (sg.sys.version)
3.8.18
PySimpleGUI Version (sg.__version__)
4.60.5
GUI Version (tkinter (sg.tclversion_detailed), PySide2, WxPython, Remi)
8.6.13
Your Experience In Months or Years (optional)
Years Python programming experience
10 Years
Years Programming experience overall
17 Years
Have used another Python GUI Framework? (tkinter, Qt, etc) (yes/no is fine)
Yes. See: https://github.com/hatfield-c/Nokia-5mm-Radio-Test-Suite
I wish I had used y'all instead.
Anything else you think would be helpful?
conda environment manager, but with the mamba package installer. program is run within conda CLI because I don't trust debuggers.
Troubleshooting
These items may solve your problem. Please check those you've done by changing - [ ] to - [X]
- Searched main docs for your problem www.PySimpleGUI.org
- Looked for Demo Programs that are similar to your goal. It is recommend you use the Demo Browser! Demos.PySimpleGUI.org
- None of your GUI code was generated by an AI algorithm like GPT
- If not tkinter - looked for Demo Programs for specific port
- For non tkinter - Looked at readme for your specific port if not PySimpleGUI (Qt, WX, Remi)
- Run your program outside of your debugger (from a command line)
- Searched through Issues (open and closed) to see if already reported Issues.PySimpleGUI.org
- Have upgraded to the latest release of PySimpleGUI on PyPI (lastest official version)
- Tried running the Development Build. Your problem may have already been fixed but not released
Detailed Description
Create a menu bar, and add some items to it.
Then go into an infinite while loop, read an image from a webcam, and then display the image using opencv. After displaying the image, you must use opencv's waitKey(0) command, which pauses the video stream and allows you to look at the image.
With the image displayed, click on the menu bar in the window, and select any option which generates an event. The program will then crash, and display the following stacktrace:
Fatal Python error: PyEval_RestoreThread: NULL tstate
Python runtime state: initialized
Current thread 0x0000242c (most recent call first):
File "z_debug.py", line 17 in <module>
I am able to gather event information from the graph and button elements without issue, and I even have my own custom buttons created using the graph drawing tools which work very well. This issue does not occur when I use python's native sleep function. Only the waitkey(0) command causes the issue.
I do not know how your's or opencv's code works under the hood, but my guess from the stacktrace is the multi-threading isn't playing nice between the two libraries. This could speak a much deeper problem within PySimpleGUI's framework and how it handles threading, at least related to the menu bar.
Code To Duplicate
A short program that isolates and demonstrates the problem (Do not paste your massive program, but instead 10-20 lines that clearly show the problem)
This pre-formatted code block is all set for you to paste in your bit of code:
import PySimpleGUI as pygui
import cv2
menu_def = [['&File', ['!&Open', '&Save::savekey', '---', '&Properties', 'E&xit']], ['&Help', ['&About...']]]
gui_layout = [
[pygui.Menu(menu_def)],
]
camera = cv2.VideoCapture(0 + cv2.CAP_DSHOW)
gui_window = pygui.Window("P4CK Calibration Suite", gui_layout, finalize = True)
while True:
_, img = camera.read()
cv2.imshow("img", img)
cv2.waitKey(0)
event, values = gui_window.read(timeout = 0, timeout_key = "__TIMEOUT__", close = False)
if event != "__TIMEOUT__":
print(event)
Screenshot, Sketch, or Drawing
Watcha Makin?
Calibration UI for distance estimation software which determines how far away something is via a video stream (it's for my PhD research). OpenCV is the most popular framework for computer vision tasks in python, and generally one needs to be able to pause the video stream through the waitKey(0) method for many use cases.
PySimpleGUI doesn't use threading under the hood. I don't see anything pointing to a crash within the PySimpleGUI code.
Have you run any of the OpenCV Demo Programs to see if they have a similar problem?
@PySimpleGUI from what I've seen the OpenCV Demo Programs only use Button elements, and not the Menu element. The button elements work fine - it's trying to read events from a Menu element that's causing the crash. Furthermore, the OpenCV Demo Programs don't use the waitKey(0) command, so this bug wouldn't appear in them anyways.
Here is some test code which uses a button instead of the menu
import PySimpleGUI as pygui
import cv2
gui_layout = [
[pygui.Button("Press me!")],
]
camera = cv2.VideoCapture(0 + cv2.CAP_DSHOW)
gui_window = pygui.Window("P4CK Calibration Suite", gui_layout, finalize = True)
while True:
_, img = camera.read()
cv2.imshow("img", img)
cv2.waitKey(0)
event, values = gui_window.read(timeout = 0, timeout_key = "__TIMEOUT__", close = False)
if event != "__TIMEOUT__":
print(event)When run, the button can be clicked on, however an event is (understandably) not processed until the cv2 window is closed by the user so as to release the blocking mechanism on the waitKey(0) command. Nonetheless the event is still generated by the click, and more importantly the application doesn't crash.
Another interesting behavior occurs when the button is clicked during a waitKey(0) call, as the the "press down" animation is not rendered in the UI window. This stands in contrast to the Menu element, which allows you to open sub-menus without issue, but crashes when you click on a menu/sub-menu item that generates an event.
It would seem that while PySimpleGUI doesn't outright instantiate threads under the hood, the tkinter library does. Hence why the menu bar instantiated by PySimpleGUI can still be interacted with even when the main thread is suspended by the waitKey(0) command. I imagine this is made even more complicated by the fact that the Menu element relies on the operatring system. There is also further evidence of this being a threading problem as demonstrated by stackoverflow posts complaining about this issue in tkinter, and they're pretty much all are related to libraries which have blocking actions (i.e. the plt.show() function from the matplotlib.pyplot library):
- https://stackoverflow.com/questions/73283716/fatal-python-error-pyeval-restorethread-null-tstate-live-data-plot-matplot
- https://stackoverflow.com/questions/52571426/tkinter-fatal-python-error-pyeval-restorethread-null-tstate
- https://stackoverflow.com/questions/7434408/fatal-python-error-pyeval-restorethread-null-tstate
Ultimately I could use a button as a workaround, but I can't get over the feeling that this all boils down to some configuration issue, like some tkinter call in the Menu element's code just needs to specify a "non-blocking" flag or something given that the other elements seem to work without crashing the application. A catastrophic error like this could be huge for a high value field - I've certainly lost hours of work before when similar errors have crashed my programs.
There are a lot of tech stacks out there based around opencv (such as ours). Having to rewrite them all to instead be rendered by the PySimpleGUI interface would be a lot of work, even if y'all's library is the superior way to define GUI's in python (which I've come to the conclusion that it is). There needs to be a solution to handle this error gracefully, or at the very least a more informative error message should be displayed to the programmer so that they can act accordingly.
I have confirmed the bug is present when trying to work with plt.show( ) as well. See the attached sample code which doesn't use any cv2 commands:
import PySimpleGUI as pygui
import matplotlib.pyplot as plt
menu_def = [['&File', ['!&Open', '&Save::savekey', '---', '&Properties', 'E&xit']], ['&Help', ['&About...']]]
gui_layout = [
[pygui.Menu(menu_def)],
]
gui_window = pygui.Window("P4CK Calibration Suite", gui_layout, finalize = True)
while True:
plt.plot([1, 2, 3, 4])
plt.ylabel('some numbers')
plt.show()
event, values = gui_window.read(timeout = 0, timeout_key = "__TIMEOUT__", close = False)
if event != "__TIMEOUT__":
print(event)To reproduce the error, perform the following steps:
- Run the attached code snippet above
- Wait until the pyplot graph is rendered
- When the PySimpleGUI window appears, click on the { File -> Save } button in the menu bar
- The application should now freeze, and the error message should be displayed in the CLI
It would seem that this bug is generalized to more than just opencv. Should I change the title of this issue to reflect that, or should I make a new issue with the condensed information?
Fatal Python error: PyEval_RestoreThread: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: initialized
Maybe it is caused by the GIL issued, and I am not sure what the detail about the GIL here.
IMO, not to use the statement cv2.waitKey(0) and the key pressed and processed by PySimpleGUI or tkinter here.
Better not to use different framework at the same time.
Example Code here.
from io import BytesIO
import PySimpleGUI as sg
import cv2
def cv2_base64(frame):
return BytesIO(cv2.imencode(".png", frame)[1]).getvalue()
camera = cv2.VideoCapture(0 + cv2.CAP_DSHOW)
success, frame = camera.read()
image = cv2_base64(frame)
menu_def = [['&File', ['!&Open', '&Save::savekey', '---', '&Properties', 'E&xit']], ['&Help', ['&About...']]]
layout = [
[sg.Menu(menu_def)],
[sg.Image(data=image, key='-IMAGE-')],
]
window = sg.Window("P4CK Calibration Suite", layout, finalize=True)
while True:
event, values = window.read()
if event == sg.WIN_CLOSED:
break
window.close()
camera.release()or capture Images by multi-thread
from io import BytesIO
import PySimpleGUI as sg
import cv2
def capture():
global running
while running:
success, frame = camera.read()
image = cv2_base64(frame)
window.write_event_value("Update", image)
def cv2_base64(frame):
return BytesIO(cv2.imencode(".png", frame)[1]).getvalue()
camera = cv2.VideoCapture(0 + cv2.CAP_DSHOW)
success, frame = camera.read()
image = cv2_base64(frame)
menu_def = [['&File', ['!&Open', '&Save::savekey', '---', '&Properties', 'E&xit']], ['&Help', ['&About...']]]
layout = [
[sg.Menu(menu_def)],
[sg.Image(data=image, key='-IMAGE-')],
]
window = sg.Window("P4CK Calibration Suite", layout, enable_close_attempted_event=True, finalize=True)
running = True
window.perform_long_operation(capture, "Done")
while True:
event, values = window.read()
if event == sg.WINDOW_CLOSE_ATTEMPTED_EVENT:
running = False
elif event == "Update":
window['-IMAGE-'].update(values[event])
if event == "Done":
break
window.close()
camera.release()should I make a new issue with the condensed information?
Please don't... this issue has plenty of info in 1 place.
I'm sure there are quite a number of interactions between tkinter and 1,000's of packages. If one of those is seen while using PySimpleGUI, that does not make it a PySimpleGUI bug. I would say the same about tkinter hitting a bug in the Python interpreter. Wouldn't be a tkinter bug.
We try to interoperate and integrate with popular packages. That's why there are dozens of demo programs using Matplotlib, OpenCV, and PIL. Not all of them are going to place nice together.
MenubarCustom
https://docs.pysimplegui.com/en/latest/documentation/module/elements/menubar_custom/
I should have added to my last comment something that would be helpful in getting you the results you're after, not just commentary on bugs.
The Menu Bar seems to have more of its implementation down in the OS. You can't change the colors of the bar itself for example. For this reason, the MenubarCustom element was created. It looks and acts like a Menu Bar, but is implemented in PySimpleGUI rather the tkinter.
Consider giving it a try instead of the normal Menu Bar.
@PySimpleGUI I just tried MenubarCustom, and I have confirmed that it works.
My apologies for not finding this sooner, but many thanks for highlighting this workaround for me - I was very worried I was going to have to code my own implementation of the menu bar using PySimpleGUI elements. I am very happy to hear that y'all already did it for me. :)
It might be worthwhile to consider adding a warning about threading to the Menu element page. Otherwise thank y'all again, and have a good rest of your evening.
It would seem there isn't a call documentation page for MenubarCustom like there is for Menu. If you click on the "MenubarCustom element call reference" link on the MenubarCustom page, it 404's:
Unfortunately the MenubarCustom element does not let you specify the color of the menu bar itself, but rather it picks for you. Instead, you can only customize the color inside the block that pops up, but that's it.
Thanks for the help anyways. I think I'm just gonna have to accept that the menu bar elements that come default with the library are more or less unusable in their current states. Even ButtonMenu is broken, as the button reverts back to the theme color when you press it. I'll figure something out with just some basic buttons instead. Which is disappointing, because the default menu bars looked really sleek and stylish.
Unfortunately the MenubarCustom element does not let you specify the color of the menu bar itself, but rather it picks for you. Instead, you can only customize the color inside the block that pops up, but that's it.
You can change the bar color.
This line of code
sg.MenubarCustom(menu_def, pad=(0, 0), k='-CUST MENUBAR-', bar_background_color='blue', bar_text_color='yellow')produced this menubar:
I noticed while researching this issue that this element is not in the call reference. We're working on adding it now. In the meantime, you can always look at the docstring in your IDE or in the source to get the same info that would be in the call reference.
def MenubarCustom(menu_definition, disabled_text_color=None, bar_font=None, font=None, tearoff=False, pad=0, p=None, background_color=None, text_color=None,
bar_background_color=None, bar_text_color=None, key=None, k=None):
"""
A custom Menubar that replaces the OS provided Menubar
Why?
Two reasons - 1. they look great (see custom titlebar) 2. if you have a custom titlebar, then you have to use a custom menubar if you want a menubar
:param menu_definition: The Menu definition specified using lists (docs explain the format)
:type menu_definition: List[List[Tuple[str, List[str]]]
:param disabled_text_color: color to use for text when item is disabled. Can be in #RRGGBB format or a color name "black"
:type disabled_text_color: (str)
:param bar_font: specifies the font family, size to be used for the chars in the bar itself
:type bar_font: (str or (str, int[, str]) or None)
:param font: specifies the font family, size to be used for the menu items
:type font: (str or (str, int[, str]) or None)
:param tearoff: if True, then can tear the menu off from the window ans use as a floating window. Very cool effect
:type tearoff: (bool)
:param pad: Amount of padding to put around element in pixels (left/right, top/bottom) or ((left, right), (top, bottom)) or an int. If an int, then it's converted into a tuple (int, int). TIP - 0 will make flush with titlebar
:type pad: (int, int) or ((int, int),(int,int)) or (int,(int,int)) or ((int, int),int) | int
:param p: Same as pad parameter. It's an alias. If EITHER of them are set, then the one that's set will be used. If BOTH are set, pad will be used
:type p: (int, int) or ((int, int),(int,int)) or (int,(int,int)) or ((int, int),int) | int
:param background_color: color to use for background of the menus that are displayed after making a section. Can be in #RRGGBB format or a color name "black". Defaults to the color of the bar text
:type background_color: (str)
:param text_color: color to use for the text of the many items in the displayed menus. Can be in #RRGGBB format or a color name "black". Defaults to the bar background
:type text_color: (str)
:param bar_background_color: color to use for the menubar. Can be in #RRGGBB format or a color name "black". Defaults to theme's button text color
:type bar_background_color: (str)
:param bar_text_color: color to use for the menu items text. Can be in #RRGGBB format or a color name "black". Defaults to theme's button background color
:type bar_text_color: (str)
:param key: Value that uniquely identifies this element from all other elements. Used when Finding an element or in return values. Must be unique to the window
:type key: str | int | tuple | object
:param k: Same as the Key. You can use either k or key. Which ever is set will be used.
:type k: str | int | tuple | object
:returns: A Column element that has a series of ButtonMenu elements
:rtype: Column
"""@PySimpleGUI thank you for providing the text documentation. My IDE is super bare-bones and can't do much beyond some basic autocomplete so I couldn't see the docstring.
I'm disappointed there isn't a way to specify the highlight color, but this will work well.
Thank you so much for your help. After working with the nightmare that is tkinter, I really love SimplePyGUI. Will definitely be purchasing a license once funding comes through.
Thank you so much for your help.
Thank you for the thank you! 
It's always nice to know when we've been of help.