`plt.plot` hangs

[Jin]

In the following example given in README, my lisp hangs for more than 1 minute when I call (pycall "plt.plot" [elided]). I could not locate the cause. But I can tell that #'import-module works well, since if I skip pycall and just do (pycall "plt.show"), it works as expected. On the other hand, I have also tried to plt.plot in a python repl; it works fine as well.

PY4CL2-CFFI> (import-module "matplotlib.pyplot" :as "plt")
T
PY4CL2-CFFI> (pycall "plt.plot"
                     (iota 10)
                     (mapcar (lambda (x) (* x x))
                             (iota 10)))
#(#<PYTHON-OBJECT :type <class 'matplotlib.lines.Line2D'>
  Line2D(_line0)
 {1006670F83}>)
PY4CL2-CFFI> (pycall "plt.show")
#<PYTHON-OBJECT :type <class 'NoneType'>
  None
 {1006672273}>

[Shubhamkar Ayare]

Can you confirm that running it straight from the REPL works fine, but running it from an emacs buffer causes lisp to hang? Can you check the output in *inferior-lisp* buffer?

I can confirm this issue and the cause seems to be simply that matplotlib does not play nice with multithreaded environments. In the future, I might provide an option to have a single-threaded python calling, so all calls to python (except perhaps the finalizers) will be made by the same thread that initiated it. This would avoid the issue.

For the time being, you can set (setf swank:*communication-style* nil) in ~/.swank.lisp and see if the issue is resolved.

[Jin]

Thanks for the input. So after setting communication-style on the fly in the lisp repl, and call ply.plot (again in the lisp repl) hangs the lisp too. Here's the message in *inferior-lisp*

2024-02-20 06:52:25.299 sbcl[34754:6081902] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'NSWindow drag regions should only be invalidated on the Main Thread!'
*** First throw call stack:
(
)
libc++abi: terminating with uncaught exception of type NSException
fatal error encountered in SBCL pid 34754 pthread 0x16f95f000:
maximum interrupt nesting depth (8) exceeded

Error opening /dev/tty: Device not configured
Welcome to LDB, a low-level debugger for the Lisp runtime environment.
ldb> 

[Shubhamkar Ayare]

So after setting communication-style on the fly in the lisp repl
...
NSWindow drag regions should only be invalidated on the Main Thread!

I'm not sure if the variable takes effect if set on the fly. Can you try setting it from ~/.swank.lisp?

[Jin]

Setting it in ~/.slynk.lisp did solve the issue. Thank you!

[Shubhamkar Ayare]

I will leave the issue open until an option is added to call python from a single thread. It doesn't look too complicated; plus, changing the swank/slynk communication style doesn't look like a great solution. We want the freedom of lisp not the restrictions of python!

[Jin]

I guess calling python from a single thread has its drawback too, right? That's why you want to make it an option, but not default. If that's the case, how would the user know ahead of hanging that if they should switch to single-thread mode?

[Shubhamkar Ayare]

I might end up making it the default; I will need to check how jupyter notebooks or other python environments behave though.

[Shubhamkar Ayare]

Recent commits up to d3df012 (and any more recent) should have added preliminary support for single-threaded mode. This is essentially an alternate system/package "py4cl2-cffi/single-threaded".

This shadows a number of exported symbols from "py4cl2-cffi" and provides their single-threaded counterparts. The intended usage is that one calls (py4cl2-cffi/single-threaded:pystart) instead of (py4cl2-cffi:pystart). Following that, function and macro calls for symbols in the package py4cl2-cffi/single-threaded would communicate to python using a single thread.

Either of the two below should work now:

(defpackage :my-package
  (:use :cl)
  (:local-nicknames (:py :py4cl2-cffi/single-threaded)))

(in-package :my-package)

;; Option 1: Use directly
(py:import-module "matplotlib.pyplot" :as "plt")
(py:pycall "plt.plot" '(1 2 3) '(1 2 3))
(py:pycall "plt.show")

;; Option 2: Use through a lisp function and package
(py:defpymodule "matplotlib.pyplot" nil :lisp-package "PLT")
(plt:plot '(2 3 4) '(32 43 23))
(plt:show)

I'll add more tests in the future.

Barring any bugs, it should be backward compatible with any exisiting use of py4cl2-cffi, so long as the first call to (pystart) happens through (py4cl2-cffi/single-threaded:pystart) instead of (py4cl2-cffi:pystart).

I'm not making this default, because this has a slight performance penalty. In addition, multithreaded usage cannot be done away completely. Lisp garbage collection relies on finalizers which, at least on SBCL, can be called through any thread.

If that's the case, how would the user know ahead of hanging that if they should switch to single-thread mode?

Currently, the only option to know this is to know beforehand how one's python packages behave in multithreaded environments.

I will need to check how jupyter notebooks or other python environments behave though.

This still remains.