python-adaptive / adaptive

:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions

Home Page:http://adaptive.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Async Running Problem with AsyncRunner

hz-xiaxz opened this issue · comments

Description:
As a newcomer to this package, I'm exploring the usage of the AsyncRunner feature, particularly aiming to employ the start_periodical_saving functionality. Below is the code snippet I've implemented:

runner = adaptive.Runner(learner, executor=executor, npoints_goal=100)
runner.start_periodic_saving(
    {"fname": f"./config/{data.nx}x{data.ny}_{data.mode}.pkl", "compress": True},
    interval=30,
)
runner.ioloop.run_until_complete(runner.task)

Since the AsyncRunner doesn't work well in a script environment, I ran it in a jupyternotebook using the %run magic command, and got the error message pointing to runner.ioloop.run_until_complete(runner.task)

RuntimeError: This event loop is already running

I suspect that the conflict arises because the runner.start_periodic_saving coroutine initiates its own event loop, which contradicts the use of run_until_complete function. Since run_until_complete is crucial for me as it ensures sequential execution of subsequent functions after runner.task completion, I'm seeking guidance on resolving this issue.

Request for Assistance:
Could you provide guidance on how to address this conflict and enable the sequential execution of functions following runner.task completion, given the necessity of using run_until_complete?

Apology for encountering another issue within such a short timeframe. Your package is truly remarkable and has significantly contributed to the success of my project. Thanks for your kind assistance

Oops. I just find a sad fact that AsyncRunner won't work for me since I'm using HDF5 library to save my data which doesn't support asyncio. h5py issue#837

Thus my problem turns out to be: will it be possible to add learner.save() feature in the BlockingRunnner class? It is necessarily supported by the asyncio feature?

The save method does actually not need to be an async method itself. It just needs to be a normal function of type Callable[[LearnerType], None].

So for example

        def save(learner):
            learner.save(**save_kwargs)

Regarding your async problems, in a script you can block with: runner.ioloop.run_until_complete(runner.task), however, in a notebook you can simply do await runner.task (docs).