Merge database using rapid

[Olivier Nadeau]

Dear devs,
While trying to use a Flow from the gallery using PhononWork, I've encountered an issue that the DDB files are only merged if you use scheduler, not if you use rapid to call tasks.

I'm trying to find a workaround to merge DDB files without using the scheduler.
I didn't find a way to create a task at the end of my PhononWork to merge files nor I find a way to artificially call on_all_ok(self).

Best regards,
Olivier

[Matteo Giantomassi]

The rapid option works as expected only if flow does not rely on callbacks
to connect the different tasks. In other words, rapid can be used only for simple flows
that do not evolve and whose dependencies are explicitly defined with the

deps = {node: "DEN}

syntax when you create the Flow.

The reason is that the single and rapid options were introduced before the
implementation of callbacks and the integration with the scheduler.

In principle, all the flows should be executed with the scheduler.
I agree that the different behaviour is confusing and I'm considering to remove
single and rapid from abirun.py.

I didn't find a way to create a task at the end of my PhononWork to merge files nor I find a way to artificially call on_all_ok(self).

The trick I use when debugging/testing is the following:
Load the flow object in Ipython with;

abirun.py FLOWDIR ipython

or alternatively with:

abiopen.py FLOWDIR/__AbinitFlow__.pickle

The abifile variable in the ipython shell is now bound to the flow object
and you can call explicitly the merge_ddb_files method

dft_work = abifile.works[1]
dft_work.merge_ddd_files()

In principle, this operation should be performed by the scheduler when on_all_ok is invoked There's indeed a protocol based on signals that must be followed to maintain the connection among the different tasks but if you know what you are doing you can decide to interact manually with the flow.

[Olivier Nadeau]

Thanks, Matteo ! I've been able to complete previous calculations using this method. I've been able to produce DDB and DVDB files using merge_ddb_files() and merge_pot1_files().

I also used abirun.py FLOWDIR/w2/t2 reset to complete the flow where w2/t2 is my task requiring DDB and DVDB.