Document interaction with starting other threads in Rust

[AF]

The following (minimal) example does not work.

use log::debug;
use pyo3::prelude::*;
use pyo3_log;
use rayon::prelude::*;

#[pymodule]
fn my_module(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
    pyo3_log::init();

    #[pyfn(m)]
    #[pyo3(name = "test")]
    fn py_test<'py>(_py: Python<'py>, n_procs: u64) {
        (0..n_procs).into_par_iter().for_each(|i| debug!("{}", i));
    }
    Ok(())
}

The problem concerns the logging macros inside parallel iterators of rayon crate.

[Michal 'vorner' Vaner]

Hello

Do you set up the logging on the python side? Can you share that part as well, please?

Also:

• Does it work outside of rayon?
• Does it work if you spawn and log in another thread (but not started by rayon)?

[AF]

Do you set up the logging on the python side? Can you share that part as well, please?

import logging

from my_module import test_rayon, test_thread

FORMAT = "%(levelname)s %(name)s %(asctime)-15s %(filename)s:%(lineno)d %(message)s"
logging.basicConfig(format=FORMAT)
logging.getLogger().setLevel(logging.DEBUG)

test_rayon(1) # works
test_thread() # does not (*)
test_rayon(2) # does not

use log::debug;
use pyo3::prelude::*;
use pyo3_log;
use rayon::prelude::*;
use std::thread;

#[pymodule]
fn my_module(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
    pyo3_log::init();

    #[pyfn(m)]
    #[pyo3(name = "test_rayon")]
    fn py_test_rayon<'py>(_py: Python<'py>, n_procs: u64) {
        (0..n_procs).into_par_iter().for_each(|i| {
            debug!("{}", i);
        });
    }

    #[pyfn(m)]
    #[pyo3(name = "test_thread")]
    fn py_test_thread<'py>(_py: Python<'py>) {
        let handler = thread::spawn(|| {
            debug!("hei",);
        });

        handler.join().unwrap();
    }
    Ok(())
}

Does it work outside of rayon?

yes!

Does it work if you spawn and log in another thread (but not started by rayon)?

no! see test_thread (*)
I think the problem is related to threads. In fact, test_rayon(1) works.

[Michal 'vorner' Vaner]

Sorry, it took me longer to get to this then I wanted originally.

It turns out the issue is not in pyo3_log (at least, not directly). You're interested in this method: https://docs.rs/pyo3/0.14.5/pyo3/prelude/struct.Python.html#method.allow_threads

A bit of explanation why. The method (py_test_thread) holds the GIL while it runs. So it starts the other thread and blocks waiting for it to complete. The other thread starts and goes to log something… but the logging internally calls into Python logging to pass the log message there, for which it needs to acquire GIL… which is being held by the first thread, so the second thread blocks on the GIL. And it nicely deadlocks.

The interaction is not completely obvious (it wasn't to me as the author, so it's naïve to assume it would be to users), so I'll keep this one open to remind me I need to put some hint into the docs somewhere.