[Core] Buffered logs lost sometimes
robertnishihara opened this issue · comments
Robert Nishihara commented
What happened + What you expected to happen
When running in IPython, some print statements are sometimes suppressed. We buffer them under the hood and deduplicate them (ignoring numerical values), but then they never seem to get printed.
In [1]: import time
In [2]: import ray
In [3]: @ray.remote
...: def f(i):
...: print(i)
...:
In [4]: [f.remote(i) for i in range(10)]
2024-05-10 18:01:35,772 INFO worker.py:1740 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8266
Out[4]:
[ObjectRef(c8ef45ccd0112571ffffffffffffffffffffffff0100000001000000),
ObjectRef(16310a0f0a45af5cffffffffffffffffffffffff0100000001000000),
ObjectRef(c2668a65bda616c1ffffffffffffffffffffffff0100000001000000),
ObjectRef(32d950ec0ccf9d2affffffffffffffffffffffff0100000001000000),
ObjectRef(e0dc174c83599034ffffffffffffffffffffffff0100000001000000),
ObjectRef(f4402ec78d3a2607ffffffffffffffffffffffff0100000001000000),
ObjectRef(f91b78d7db9a6593ffffffffffffffffffffffff0100000001000000),
ObjectRef(82891771158d68c1ffffffffffffffffffffffff0100000001000000),
ObjectRef(8849b62d89cb30f9ffffffffffffffffffffffff0100000001000000),
ObjectRef(80e22aed7718a125ffffffffffffffffffffffff0100000001000000)]
(f pid=31071) 7
In [5]: time.sleep(1)
In [6]: time.sleep(5)
In [7]: ray.put(1)
Out[7]: ObjectRef(00ffffffffffffffffffffffffffffffffffffff0100000001e1f505)
In [8]:
In [8]: f.remote(0)
Out[8]: ObjectRef(359ec6ce30d3ca2dffffffffffffffffffffffff0100000001000000)
(f pid=31068) 0
In [9]:
In [9]:
Do you really want to exit ([y]/n)?
Versions / Dependencies
Python 3.11.4
ray, version 2.21.0
macOS 13.5
Reproduction script
See above
Issue Severity
None