Python: get() method not releasing GIL

[Arne Welzel]

I think this came up in Slack before. When calling subscriber.get() and blocking, other Python threads do not continue to run. The broker bindings should release the Python GIL before blocking.

https://pybind11.readthedocs.io/en/stable/advanced/misc.html#global-interpreter-lock-gil

import threading
import time

import broker

def poll():
    while True:
        print(time.time(), "thread polling")
        time.sleep(0.3)

def main():
    t = threading.Thread(target=poll, daemon=True)
    t.start()
    time.sleep(2)

    ep = broker.Endpoint()
    ep.listen("127.0.0.1", 60000)
    with ep.make_status_subscriber(True) as ss:
        while True:
            print(time.time(), "ep.get() blocks thread from running")
            status = ss.get()

            print(time.time(), "got a status")
            time.sleep(1)

if __name__ == '__main__':
    main()

The output is as follows, showing how the thread polling stops once ep.get() is called. Connecting another peer shortly allows to do more polling as there's a time.sleep() which yields the GIL when sleeping.

$ PYTHONPATH=.:/opt/zeek-dev-prod/lib/zeek/python python3 threaded.py 
1679943976.7454154 thread polling
1679943977.0458524 thread polling
1679943977.3463733 thread polling
1679943977.6468718 thread polling
1679943977.947265 thread polling
1679943978.2476246 thread polling
1679943978.548081 thread polling
1679943978.7587862 ep.get() blocks thread from running
1679943983.725927 thread polling
1679943983.7260008 got a status
1679943984.0264978 thread polling
1679943984.327149 thread polling
1679943984.6278195 thread polling
1679943984.727265 ep.get() blocks thread from running
1679943984.727514 got a status
1679943984.928495 thread polling
1679943985.2289653 thread polling
1679943985.5294564 thread polling
1679943985.7281764 ep.get() blocks thread from running


^Cterminate called without an active exception
Aborted (core dumped)