Python->Java call does not see value changes from different thread

[John Collins]

I've been looking through the documentation and have found nothing on this. I'm using Java 11, trying to keep the Python side simple and single-threaded, and trying to avoid Java->Python callbacks. So Python makes a call to Java that starts a process that involves communication with another process, which may be local or remote. On the Java side, this first call starts a Java thread that initiates communication, and once communication is established it calls notifyAll() on a monitor object. So that returns to the Python side immediately, once the Java thread has started. Then Python makes another call to a Java method that uses wait() on the same monitor object. This process has worked reliably for years when the "client" is in Java, but with a simple proof-of-concept Python client using Py4j, that second call never returns.

[John Collins]

I suspect my problem is not with threading. I wrote a simple Java example that has essentially the same interaction with Python, in which Python calls a method that starts a process, and then calls another method that blocks and waits for the process to complete. The process in this case is Thread.sleep(). Here's the Java example, the Java test case, and the Python test case. All of them work correctly.

So I am not sure where to look next to resolve this problem. Another thing that happens in the real Java code is that it logs in to an Apache Active MQ server and receives a few hundred messages, one of which is the one we are waiting for on the Python side.

[John Collins]

A little more info that might shed some light; it makes me suspect there's something weird going on in the thread used by the Java gateway. Here's a bit of Java code:

public class ContextManagerService {
  private boolean started;

  // called from message queue thread pool
  public void handleMessage (SimStart ss)
  {
    log.info("SimStart");
    started = true;
    log.info("After SimStart started = {}", started);
  }

  // called from Python
  public boolean checkForStart ()
  {
    log.info("checkForStart {}", started);
    return started;
  }
}

In the log I see

17910 INFO  core.BrokerMessageReceiver: received message:
<sim-start>
  <start>
    <iMillis>1626878073456</iMillis>
  </start>
</sim-start>
17911 INFO  core.PowerTacBroker: SimStart - start time is 2021-07-21T14:34:33.456Z
17911 INFO  samplebroker.ContextManagerService: SimStart
17911 INFO  samplebroker.ContextManagerService: After SimStart started = true
17991 INFO  samplebroker.ContextManagerService: checkForStart false
18091 INFO  samplebroker.ContextManagerService: checkForStart false
18192 INFO  samplebroker.ContextManagerService: checkForStart false

The entries at 17910-17911 are from the message queue thread. The entries starting at 17991 are the gateway thread, which is getting called every 100 msec. I'm attempting to use busy-waiting rather than wait/notify to communicate this info to the Python side, but a private class variable is not seen by the gateway thread. A unit test written in Java works as expected.

The started variable is a boolean, it is not thread-local, so updates are atomic and synchronization should not be needed. I tried it with synchronization, also works in Java and not with the Python gateway.

[John Collins]

Problem solved -- I had forgotten that Spring services are not necessarily singletons. and Python was looking at a different instance than the one Spring created.