flush output of %%px --no-stream on error

[Francesco Ballarin]

Hi,
I have the following notebook with ipyparallel 8.1.0.

# Cell 1: setup cluster
import ipyparallel as ipp

cluster = ipp.Cluster(engines="MPI", profile="mpi", n=2)
cluster.start_and_connect_sync()

# Cell 2: run with %%px a code that raises an error
%%px
import time

print("Hello, World!")
time.sleep(3)

raise RuntimeError("Stop here")

# Cell 3: run with %%px --no-stream a code that raises an error
%%px --no-stream
import time

print("Hello, World!")
time.sleep(3)

raise RuntimeError("Stop here")

The output I see in the jupyter notebook when running cell 2 is

[stdout:0] Hello, World!
[stdout:1] Hello, World!
%px: 100%
2/2 [00:01<00:00, 1.98tasks/s]
[0:execute]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_12510/2436914339.py in <module>
      4 time.sleep(3)
      5 
----> 6 raise RuntimeError("Stop here")

RuntimeError: Stop here
[1:execute]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_12511/2436914339.py in <module>
      4 time.sleep(3)
      5 
----> 6 raise RuntimeError("Stop here")

RuntimeError: Stop here
2 errors

where, as expected, the two lines printing Hello, World! appear three seconds before the error is raised.

However, when running cell 3, the only output I get is

%px: 100%
2/2 [00:01<00:00, 1.89tasks/s]
[-1:execute]:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_12510/2436914339.py in <module>
      4 time.sleep(3)
      5 
----> 6 raise RuntimeError("Stop here")

RuntimeError: Stop here

[-1:execute]:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_12511/2436914339.py in <module>
      4 time.sleep(3)
      5 
----> 6 raise RuntimeError("Stop here")

RuntimeError: Stop here

which:

• is lacking the output of print. Is there a way to flush the output before the error is raised? I tried patching my installation by duplicating this line in try/except blocks before raising, but output still did not appear.
• why is the process number equal to -1, rather than 0 and 1 respectively?

Thanks!

[Min RK]

Thanks for the test cases! Should be fixed by #663.

[Francesco Ballarin]

Hi @minrk , thanks for the feedback and the PR in #663!

I re-installed ipyparallel with pip3 install git+https://github.com/minrk/ipyparallel.git@fix-engine-id and manually confirmed the version change (8.2.0.dev and not 8.1.0 anymore) and that the changes from the branch are indeed there.
However, I still get the same output for cell 3 as in the original post. Should I do anything else after I update ipyparallel to see the effect of the changes you made? (Maybe clear the profile_mpi in $HOME/.ipython?)

Thanks!

[Min RK]

The fix for -1 is in the engines, so my suspicion is that the engines themselves are not getting the update. Can you check ipyparallel.__version__ and ipyparallel.__file__ in the engines, too (%px ipyparallel.__version__)?

[Francesco Ballarin]

I think that the engines got the update. This is the current content of my test notebook, with the third cell to verify that the engines are on the #663 branch

import ipyparallel

cluster = ipyparallel.Cluster(engines="MPI", profile="mpi", n=2)
cluster.start_and_connect_sync()

%%px
import ipyparallel
import os
magics_py = os.path.join(os.path.dirname(ipyparallel.__file__), "client", "magics.py")
assert os.path.isfile(magics_py)
# Check that at least one change from #663 is available on the engine
diff = "self.shell.user_ns[args.save_name] = self.last_result"
with open(magics_py, "r") as f:
    assert diff in f.read()  # Does not fail, so the changes from #663 seem to be there

%%px --no-stream
import time

print("Hello, World!")
time.sleep(3)

raise RuntimeError("Stop here")
# Same stdout as the opening post

[Min RK]

OK, thanks for testing! Looks like there must be a place I missed.

[Min RK]

Indeed, I fixed it for apply, but missed it for execute. Should be fixed, now

[Francesco Ballarin]

Thanks, I reinstalled with the updated #663.

I confirm that the engine id is now 0 and 1, rather than -1, so #663 indeed fixes the second item in my opening post. However, the first item still seems to be an open issue, as I cannot see the Hello, World! print out.

[Min RK]

Got it. Can you try one more time?

[Francesco Ballarin]

Thanks @minrk ! I confirm that both issues are now fixed by #663.

[Min RK]

Wonderful, thanks for testing!