Drop synchronous counterparts?

[ItsDrike]

Currently, mcproto provides both asynchronous and synchronous versions for pretty much everything that needs it.

Necessity of synchronous code

In general, people that already have a synchronous code-base won't actually be that disadvantaged if sync support is dropped, moving to purely async model. This is because they can always just use asyncio.run in their synchronous code, which will block until the function is complete, essentially acting synchronously, and only write async functions for the code working with mcproto.

The only reason synchronous code might be beneficial is to make it easier to use the library for people who don't have that much experience with async code. Indeed, a purely async library might discourage some less experienced people from even trying it out, however, I'd say this library is already relatively complex and will require users to have a certain skill level, at which async likely won't be that "mysterious" and hard to understand. Nevertheless, it might discourage some users from using mcproto.

But it's also important to consider the other side, dropping synchronous versions could be beneficial even to the less experienced people, as they won't accidentally end up using the wrong (synchronous) versions, blocking the rest of their async code. This is actually somewhat common with discord.py users (from what I've seen in people that use mcproto), and simply being purely async would avoid such cases completely.

Code duplication

This effort to support both versions already causes some major code duplication in the codebase, with for example the protocol/base_io.py file holding essentially exact duplicates between the BaseAsyncWriter and BaseSyncWriter classes, and also between BaseAsyncReader and BaseSyncReader, which then carries over to duplication in connection.py, with the base classes SyncConnection and AsyncConnection, and then all subclasses moving from those (TCPAsyncConnection and TCPSyncConnection, and similarly for UDP). Another place affected is packets/interactions.py, which needs alternatives for both of these approaches in the reader/writer functions for packets (sync_write_packet and async_write_packet, similarly for readers). Eventually, this will also end up affecting any HTTP requests we might end up making (such as for interactions with mojang API, for yggdrasil authentication, but also potentially for OAuth2 based Microsoft authentication).

In almost all of these places, the alternative code is just the same rewritten code, with an extra await and a change in a function name or type-hint. That means any changes made to one of those will need to be copied to the other, including docstring changes.

This code duplication is then also affecting the tests, which currently either have the same full code duplication as in the codebase, or use some complex mechanics to "synchronize" the code, allowing us to only write tests for the synchronous classes, with the async ones getting wrapped to act as the same, synchronous counterparts, with the help of asyncio.run. This can make many people confused about how to write new tests, and even how to understand and edit the existing ones.

[Martin K]

That does make using the library synchronously annoying, but if it gets rid of a lot of code duplication - I'd say it's worth it.

[CoolCat467]

I would support having async only as well, but one thing I forsee issues with is the current way Buffer is set up. More so, since Buffer is subclassing bytearray, it would be a bit interesting working with it where the reading and writing is asynchronous, but everything built in in bytearray is synchronous.

[ItsDrike]

Hmm, that's actually a great point which I haven't considered, thanks for bringing it up @CoolCat467!

Indeed the Buffer class should probably not become asynchronous as that is pretty misleading on what it's duing. An asnyc function should pretty much always indicate that there will be some I/O operation, during which we can actually switch to another task in the event loop while we're waiting on this operation to finish. Clearly, this is not what buffers do, and I'm not sure how I feel about making those async.

That said, the synchronous code is a burden to maintain here, so I'm not quite sure how to resolve this. We could drop async support only in some places, most notably the connection classes, leaving the rest as-is, however that wouldn't actually be all that helpful in removing this burden, as most of the code duplication is in the protocol/base_io.py file, which would need to remain unchanged if we want buffer class to be synchronous.

I will say that I feel pretty strongly against bringing in synchronous code into #129, handling HTTP requests and communications with various APIs.

[CoolCat467]

I have an idea, but I don't like some of the consequences of it. My idea is that we could write a class decorator or something that would find async functions and provide a sync counterpart that would just be a wrapper for starting an event loop and executing the async code. It would work, but it would introduce a lot of overhead and it would also lead to compatibility issues for working with different asychronous libraries like Trio.

Edit: An even better idea would be doing something like writing a macro that would take a file with async functions and remove the async and await keywords, and then you'd have a synchronous version that was auto-generated

[ItsDrike]

This actually already exists in mcproto, but is only utilized in tests, to make it eaiser to test both async and sync implementation of the readers and writers. There is the SynchronizedMixin class, and a synchronize decorator.

The reason why this isn't used outside of tests is because it completely breaks static analysis due to how dynamic it is. And indeed, it also comes with a processing time overhead. One of the goals of mcproto is to be fully statically typed, and this solution would break that goal.

We could probably write stub files that would contain the actual typing information, but at that point, we might as well just go with the current code duplication approach, as with stubs, we'd just end up manually writing the signatures for all of the functions that would be dynamically generated, cancelling the benefits of that dynamic generation.

Edit: An even better idea would be doing something like writing a macro that would take a file with async functions and remove the async and await keywords, and then you'd have a synchronous version that was auto-generated

This is an interesting idea, but it doesn't really solve everything, and would probably be way too complicated to actually implement. Consider for example the connection classes, where asyncio is utilized to create the underlying connection (asyncio.create_connection), you can't simply remove the await here and expect things to work synchronously, which makes this approach only viable in some limited scenarios. It would probably work with the reader/writer classes in base_io, but introducing this kind of complex system just to cover these cases seems like a huge overkill.

[CoolCat467]

I agree about the create_connection part, but if the intent is to only have synchronous stuff for Buffer, then it shouldn't be too bad. There are some great modules for modifying python AST, and with structural pattern matching it might not be super hard

[ItsDrike]

Indeed, if we do this purely to make the buffer work, it should be alright. So now the question is whether doing something like this would actually be a good idea, as opposed to purely whether it would be viable.

There is a few things that I don't like about introducing this kind of system:

Yet another thing to learn

I've recently been trying to reduce the burden of contributing to this repo, as it was (and still is) using quite a lot of various tools already, which require the contributors to understand these (i.e. the tools used for static analysis, formatting, documentation creation (sphinx) and changelog keeping tool (towncrier)). Therefore I'm not sure I like the idea of bringing yet another complex system like this.

This system would mean that every contributor would need to be made aware of it's existence, and would require people to run this code generation script any time a change to the async base io classes was made. This is not trivial to explain and it would be pretty difficult to automate (I talk about why that is below), so it would certainly decrease the "ease of contributing" even more.

Another thing to consider is that this makes it impossible to use any async calls in the base io async classes, other than when referencing internal async methods, as code generation like this woudln't be able to account for an async call made to an external library (such as the discussed asyncio.open_connection, but this also goes for any internal async functions that are not just a method of the same class, meaning it would just become synchronous). However I'd say the need to do something like this in the base io classes is very unlikely, so it's probably not an issue.

Automation?

First of all, we should establish that the sync file can't simply be generated only during a release. That is because pyright would be failing anywhere where the sync classes would be used (so, mainly in the buffer class). This means we will need to keep the generated file around constantly, as if it was just another regular python file in the code-base.

However this immediately means that contributors will need to be aware that this file is generated, and that they shouldn't be making any direct changes to it, and instead leaving it to the (theoretically automated) code generation system. So there's already a reason why this can't be completely seamless with contributors not having to learn anything new, even when this system would be automated.

Automation approaches, and their issues

OK, so let's consider how this could actually be automated now. With the way we currently have things set up, there are 2 ways automation could be done:

• GitHub Workflows

  With GitHub workflows, this would mean having a CI action that could detect changes made to async base io, and run the code generation script, automatically creating a new commit in that PR. I don't like this for 2 reasons.

  1. Ceating new commits automatically can be distracting, and unexpected. The contributors might simply push their code once they finished a feature, and then immediately start working on another part of the PR, not looking at what happened on GitHub. Then, after they finished that, and they attempt to push again, suddenly, github will refuse this push, as there is already a new commit in there, forcing you to first do a git pull.
  2. Automatically pushing code to forks might even be possible. When a fork repo is made, GitHub will by default allow maintainers of the original repo (people with write access to mcproto) to push changes to the fork repo. This would indeed also allow bots to push new commits, however this feature can be disabled by the fork owner, at which point this CI workflow would fail.

  
  Another option with GitHub workflows would be to simply enforce that the user has ran this script, if there was a change done made to the async file. But I wouldn't count that as automation of code generation, that's just verification, and it would still require the user to run the script manually, which is the annoyance we're trying to automate.
• Pre-Commit hook

  With pre-commit, you could indeed add a hook that runs each time, and even set it up in a way that can easily detect chanegs to the base io file, only triggering the code generation script when that happens. But it's not without issues either:

  1. It's yet another hook, slowing down committing. It's already pretty annoying to have to wait for pre-commit each time you create a new commit, a script like this would slow things down even more. But that's not that huge of an issue.
  2. Contributors aren't technically forced to use pre-commit. They don't have to install the pre-commit hook at all, or they could just make a commit with --no-verify, skipping pre-commit. When that happens, we could end up with inconsistent code between the async and the generated sync versions of the code-base. This could be something left for manual review, leaving it for maintainers as another thing they should be looking for when reviewing.

Fundamental automaton issue

Regardless of the specific issues these automation approaches, there is a fundamental issue, comming from the fact that we need the generated file to be constantly present and up-to-date for static analysis. That is the simple fact, that any kind of automation will only run so often. In both of the cases above, the code generation would only run after or during commiting.

To explain why this is an issue, consider this scenario: You've just made a change to the write_optional function, which now takes an additional required argument. You made this change only in the async version, as you know the sync one will simply be generated. Pyright will now quickly inform you of all of the places in the code-base, where the write_optional was used that doesn't match this new signature, so you can quickly change those, and make that a part of your single commit. Now, after you fixed all of the issues from pyright, you decide to actually commit. However the moment you do, the code-generation script is ran, altering the (previously unchanged) synchronous version of the base io classes, making that adjustment to write_optional carry over to the sync classes. Suddenly, pyright detects a whole bunch of new issues with the places where write_optional was used from the synchronous code-base, that it previously didn't see, because the generated file wasn't updated yet.

With pre-commit, you will actually fail to create the commit here, assuming pyright ran after the code generation, which at least forces you to address the changes first, which is certainly better than with the GitHub CI option, but it's still not ideal and it slows the development down. Also, again, users can skip pre-commit, meaning we would need GitHub CI to at least verify that the codegen script was ran, and the sync file is "up to date" with the async one.

Conclusion

It's clear that automation would work poorly at best, and would still require the contributors to have some knowledge that there is code-generation going on, forcing them to adjust their workflow to respect that. This will therefore slow down development yet again, and make it even harder to contribute than it already is.

I'm therefore not really convinced that this is something worth doing, as it might prove to be even more annoying to deal with than the code duplication caused by sync code.

However I don't really have any good alternatives, other than simply keeping both sync and async versions around, or to make the very weird choice of making buffer interactions asynchronous, which I'm really not a big fan of either.

Edit:

A bit about why making buffer async is a bad idea

To provide a better explanation, other than "I'm not a big fan of" for why buffer should not be made asynchronous.

First of all, as I've mentioned before, an async function implies that there is a blocking I/O operation that will need to be waited for, during which we can switch to something else in the event loop. Clearly though, this is not what's going on in the Buffer class, all that occurs during write or read call is the simple mutation of our buffer class.

Additionally though, Buffer inherits from bytearray, and this means it already includes some methods such as extend, which are synchronous. That means Buffer wouldn't even be purely async, and would instead become a really weird mix of being half-async, half-sync. This is really not ideal, considering Buffer is, and always will be a part of the public API of mcproto, meaning the library users will be able to use it directly. Exposing Buffer publically with the interactions being so inconsistent in clearly this way is a bad idea, and this is why I'm so fundamentally against it.

[ItsDrike]

Alright, what about leaving the current synchronous alternatives as they are, but leave some features, especially when it comes to HTTP communication (minecraft APIs) purely async, and just mentioning this in the docs.

This would make most people who would want to actually make a bot capable of reaching play state (logging in) have to use async at least for these calls, likely making them use async connections as well, though not forcing them to, and projects that would not need these API calls could remain fully synchronous (only useful for offline (aka "warez") servers, or for things that don't need to go through the login state, i.e. a lib that only obtains the server's status, such as mcstatus)

As the sync support is already in place, while it may be a bit hard to orient in, it's unlikely to actually need that many updates, as it's based on how the core communication happens in the minecraft protocol. Minecraft isn't likely to change this between protocol versions, as it would be a completely breaking change, making it impossible for people on this version to potentially even obtain server status from older minecraft version. Because of that, even though it is a significant source of code repetition, it probably won't be a huge burden to maintain.

I don't love the idea of this split, where there are some sync alternatives, but not for everything. However as I said, the base io and connection classes are unlikely to need much maintaining, and the newer things that don't necessarily need the support (that don't need to utilize the conneciton classes for example) can remain async only.

[CoolCat467]

That sounds like a pretty good plan.

Slightly unrelated, but will there be support for different asynchronous libraries like Trio? It's not a huge issue, because you can run an asyncio event loop on top of Trio, but it would be nice to have native support for Trio.

[ItsDrike]

Slightly unrelated, but will there be support for different asynchronous libraries like Trio? It's not a huge issue, because you can run an asyncio event loop on top of Trio, but it would be nice to have native support for Trio.

That might require a rework of the Connection classes, as they use asycnio.open_connection, I'm not sure how Trio handles connections. The rest of the things should probably be compatible? I don't have like any experience with Trio, but there's nothing else asyncio specific that I can think of in the code-base. There are some asyncio.wait_for calls, and we're also using asyncio_dgram, for UDP which from the name sounds pretty asyncio specific, but I don't know how it works internally. Probably worth making another issue about it though. But as I have no experience with it here, you would probably need to implement that yourself, if approved.