Callbacks return too early?

[the-vindicar]

It appears that most coroutine methods of Client don't wait for the action to be completed. A simple example:

import asyncio
import pydle

_IRC = pydle.featurize(pydle.features.RFC1459Support, pydle.features.CTCPSupport, pydle.features.AccountSupport)
class IRC(_IRC):
    async def on_connect(self):
        print('2. on_connect')
        await self.join('#test')
        print('4. on_connect exiting')
    
    async def on_join(self, channel, user):
        print('3.',user,'joined',channel)
        await self.message(channel, 'hi')

async def main():
    irc = IRC('Bot', [], 'bot', 'Bot')
    print('1. connecting')
    await irc.connect('192.168.0.2', 6667, reconnect=False)
    print('5. returned from connect()')
    await asyncio.sleep(5.0)
    await irc.quit('bye')

asyncio.run(main())

I get this behaviour

1. connecting
5. returned from connect()
2. on_connect
4. on_connect exiting
3. Bot joined #test

while expecting this behaviour:

1. connecting
2. on_connect
3. Bot joined #test
4. on_connect exiting
5. returned from connect()

Is there a way to get the expected behaviour? I understand that a long-running task in a callback would effectively stop the bot, but spawning a new task is much easier than waiting for a callback to fire.

[Joshua Salzedo]

Thanks for the report!

Based on what I can see async def connect(...) does not wait for the on_connect handler to fire; instead it sets up the connection and spawns pydle's IRC handler in a new task.

I think it would be possible to achieve the desired effect using Pydle's feature system to overload the async def connect(...) and extend async def on_connect.

in connect one would need to spawn a Task and store it on the instance, await the superclass impl, then await the completion of the spawned task.
in on_connect one would need to complete the task spawned in connect.

If this solution works for you, I can work up a PR for this.

It seems async def join() doesn't await for a future to be completed, although joining a channel before the connection is completed seems illogical to me.

import asyncio
import pydle

_IRC = pydle.featurize(pydle.features.RFC1459Support, pydle.features.CTCPSupport,
                       pydle.features.AccountSupport)


class IRC(_IRC):
    def __init__(self, *args, **kwargs):
        super(IRC, self).__init__(*args, **kwargs)
        # create an event that will be set once on_complete is called
        self.connection_event = asyncio.Event()

    async def on_connect(self):
        self.connection_event.set()

        print('2. on_connect')
        await self.join('#test')
        print('4. on_connect exiting')

    async def on_join(self, channel, user):
        print('3.', user, 'joined', channel)
        await self.message(channel, 'hi')

    async def connect(self, *args, **kwargs):
        await super(IRC, self).connect(*args, **kwargs)
        await self.connection_event.wait()


async def main():
    irc = IRC('Bot', [], 'bot', 'Bot')
    print('1. connecting')
    await irc.connect('127.0.0.1', 6667, reconnect=False)
    print('5. returned from connect()')
    await asyncio.sleep(5.0)
    await irc.quit('bye')


asyncio.run(main())

1. connecting
2. on_connect
4. on_connect exiting
5. returned from connect()
3. Bot joined #test
Unknown command: [irc.unknown.com] 5 ['Bot', 'UHNAMES', 'NAMESX', 'SAFELIST', 'HCN', 'MAXCHANNELS=10', 'CHANLIMIT=#:10', 'MAXLIST=b:60,e:60,I:60', 'MAXNICKLEN=30', 'NICKLEN=30', 'CHANNELLEN=32', 'TOPICLEN=307', 'KICKLEN=307', 'AWAYLEN=307', 'are supported by this server']
Unknown command: [irc.unknown.com] 5 ['Bot', 'MAXTARGETS=20', 'WALLCHOPS', 'WATCH=128', 'WATCHOPTS=A', 'SILENCE=15', 'MODES=12', 'CHANTYPES=#', 'PREFIX=(qaohv)~&@%+', 'CHANMODES=beI,kLf,l,psmntirzMQNRTOVKDdGPZSCc', 'NETWORK=unknownNet(tm)', 'CASEMAPPING=ascii', 'EXTBAN=~,tmTSOcaRrnqj', 'ELIST=MNUCT', 'are supported by this server']
Unknown command: [irc.unknown.com] 5 ['Bot', 'STATUSMSG=~&@%+', 'EXCEPTS', 'INVEX', 'CMDS=USERIP,STARTTLS,KNOCK,DCCALLOW,MAP', 'are supported by this server']
Unknown command: [irc.unknown.com] 396 ['Bot', 'Clk-63E1DF0D.local', 'is now your displayed host']

Process finished with exit code 0

[the-vindicar]

Thank you for a quick response, @theunkn0wn1 !
If I understood your suggestion correctly, it's almost the same as creating an asyncio.Event before calling a method, and setting it once the callback is called (after the necessary checks).
But if I get this right, you will end up with quite a large amount of flags (be those tasks, events, futures or what not). Things will get worse for actions like joining/parting, as you will also have to check which channel should have its corresponding task completed.
Though it seems like it could work, in the end.

I'm wondering because I'm trying to plug pydle into an asynchronous project of my own via an adapter class, and I keep getting a situation where loop somehow ends up terminating just before main() completes. My own quick and dirty implementation of IRC protocol doesn't have this problem when plugged through the same adapter and ran by the same test driver code (which does the same thing as one I posted above). So far the matter of waiting for the action to be completed seems to be the only important difference between the two implementations (mine and pydle-based).

[Joshua Salzedo]

Pydle shouldn't be prematurely terminating the event loop, it should continue running as long as the event loop lives (or you explicitly call .disconnect).

As a suggestion, please try explicitly passing Pydle a reference to the event loop. By default, Pydle will acquire its own loop handle and I am not sure if that may be causing problems in your context.

client = Client(..., eventloop=<handle to loop>))

---

But if I get this right, you will end up with quite a large amount of flags (be those tasks, events, futures or what not). Things will get worse for actions like joining/parting, as you will also have to check which channel should have its corresponding task completed.

All of this would optimally be abstracted away by the IRC library; i know Pydle already does this for IRC commands like WHOIS.
Logically, it wouldn't be too difficult to extend Pydle (PRs welcome) to add this tracking to other places its apparently missing, like join and part.

[the-vindicar]

Yeah, I explicitly call .quit() at the end. It's likely that I misunderstand something, so I won't bother you with it. =)
I'm not sure I grok pydle enough to write up a PR off the bat, but I imagine something along the lines of...
Dict[Tuple[str, str], Tuple[Future, List[str]]]
Basically, a cache. Key is event code (JOIN, NAMES, etc) and an additional parameter. Value contains a future and an accumulator for information gathered along the way.
Thus, action methods like .join() and some event handlers like on_whois will add a value to the cache, and return an associated future.
Other event handlers will generate the key and check its existence in the cache, then either:
a) add data to the cache (like multiple NAMES or WHOIS lines)
b) remove the future-value pair from the cache, and complete the future using the value (end of NAMES list, for example).
c) remove the future-value pair from the cache, and fail the future using an appropriate exception.
d) do nothing because somehow the key is not in the cache.

I think I will go and add a system like that my pydle adapter to see if it works...

[Joshua Salzedo]

Yeah, I explicitly call .quit() at the end. It's likely that I misunderstand something, so I won't bother you with it. =)

Note that if pydle isn't provided with an event loop, it will close its own loop on quit. It will not do this if the loop is externally provided.

a) add data to the cache (like multiple NAMES or WHOIS lines)

this feature is already present; pydle's WHOIS handlers update Pydle's internal user state upon receipt. the whois method also async-blocks until the response is received. In my downstream project, i make sure to do a whois against a user before using pydle's user dict to ensure that user is up-to-date.

for join to behave as expected (not return early like connect), it will need to create a future that on_join completes.
this can probably be done via some dict dict[str, Future] where the str is the channel being targeted.

something to the effect of, and this is psudocode:

async def _on_connect(...):
    ...
    future = self.join_futures.get(joined_channel)
    if future:
        future.set_result(true)

[the-vindicar]

Yeah, this is basically what I meant - I just tried to generalize the system, because WHOIS effectively does the same thing, but separately. It's a matter of preference, though.

Note that if pydle isn't provided with an event loop, it will close its own loop on quit. It will not do this if the loop is externally provided.

Now that is something I didn't expect... thanks, I will see if giving it a loop explicitly changes anything.

EDIT: Yep. That was the thing.

[Shiz]

On 11 Aug 3 Reiwa, at 21:51, Joshua Salzedo ***@***.***> wrote: Yeah, I explicitly call .quit() at the end. It's likely that I misunderstand something, so I won't bother you with it. =) Note that if pydle isn't provided with an event loop, it will close its own loop on quit. It will not do this if the loop is externally provided. a) add data to the cache (like multiple NAMES or WHOIS lines) this feature is already present; pydle's WHOIS handlers update Pydle's internal user state upon receipt. In my downstream project, i make sure to do a whois against a user before using pydle's user dict to ensure that user is up-to-date. for join to behave as expected (not return early like connect), it will need to create a future that on_join completes. this can probably be done via some dict dict[str, Future] where the str is the channel being targeted.

pydle not waiting for joins (or similarly, a bunch of IRC commands) to complete is by design, and is related to the asynchronous nature of IRC itself. There is no straightforward way to know if a join to a channel was successful: it can fail for any number of reasons, some documented in the original spec (e.g. channel is invite-only), some undocumented by even common standards. If the server refuses to let the client join a channel, it will (generally, but I don’t think even that is strictly required!) send a reply message (RPL_*) with the error code, but if the client is unaware of that specific reply code existing, it will not interpret it as related to the pending channel join. As such, the future will never be resolved, and on_connect() will hang forever. This is not a problem per se (pydle dispatches every message to a new job in the event queue), but it does mean that any code you wanted to run after in the same handler will never run. This is why the example code makes use of the `on_join` handler that much instead of awaiting a future in on_connect(): it's better to run some other code when you *know* you have joined rather than potentially awaiting a state that may never happen, nor be verifiable that it failed. Which is to say all of this is technically possible, but it’s designed the way it is for a reason. It’s at least worth thinking about. :)

[Joshua Salzedo]

Thanks for the perspective @shizmob <3

I also agree its best to hook on on_connect, and this is also the pattern seen in discord.py.

The best this library can do is be standard conforming; if the standards don't provide a reliable mechanism for reporting join failures there isn't a whole lot we can do.

[the-vindicar]

I understand and agree. This issue can be considered closed then, especially since the reason for me opening it turned out to be completely unrelated.