AutoReconnect handled by paho instead of consumers.
ZekeButlr opened this issue · comments
Is your feature request related to a problem? Please describe.
I've noticed that handling reconnection logic in case of MQTT connection loss is currently a responsibility placed on the consumers of the paho library. This often leads to repeated code across different projects and potential inconsistencies in how reconnections are handled. I'm always frustrated when I have to implement and maintain reconnection logic in each project that uses paho, instead of having a standardized, reliable solution directly integrated into the library.
Describe the solution you'd like
I propose the integration of an AutoReconnect feature within the paho library itself. This feature would manage reconnection attempts automatically when a connection to the MQTT broker is lost. Key aspects of this feature would include configurable parameters such as MaxRetries, RetryInterval, MaxRetryInterval, and BackoffFactor. Additionally, a ReconnectHandler callback could be implemented to provide a new connection and connection configuration for each reconnection attempt. This integrated approach would ensure a consistent and efficient handling of reconnections across all projects using the library.
Describe alternatives you've considered
An alternative could be to provide a standard, separate package or toolkit alongside paho that specializes in handling reconnections. While this would still centralize the reconnection logic, it would require additional integration from the consumers. Another option is to enhance the documentation with best practices and sample implementations of reconnection logic, but this would not alleviate the need for each project to implement its own solution.
Additional context
Integrating AutoReconnect into paho would significantly enhance the robustness and ease of use of the library. It would reduce the boilerplate code and complexity on the consumer's side, allowing developers to focus more on the core functionality of their applications. This feature would be particularly beneficial in environments with unstable network conditions, where frequent reconnections are necessary.
Closed PR as a first attempt to resolve this issue: #208
CC: @MattBrittan
Thanks @ZekeButlr.
The v3 client worked this way; it's reconnection logic is similar to what you are proposing, our experience was that it's very difficult to get this right (it's taken years to resolve deadlocks/races etc).
The design of paho.golang/paho, is in many ways, a response to the complexity of the V3 client. The V3 client tried to be all things to all people, and the result was that it's over complex and has, historically, suffered from bugs mainly due to this complexity (I think most have been fixed now!).
paho.golang/paho is designed to accept a net.Conn and effectively terminate when that connection is lost. This design has led to assumptions within the code that will make adding automatic reconnection into paho.golang/paho itself difficult (a fair number of design decisions would need to be revisited; it would not be a simple change).
However I agree that reconnection logic is a common requirement and is often implemented poorly. This is why paho.golang/autopaho exists; as per the readme:
autopaho (import "github.com/eclipse/paho.golang/autopaho") is a fairly simple wrapper that automates the connection process and will automatically reconnect should the connection drop. For many users this package will provide a simple way to connect and publish/subscribe as well as demonstrating how to use the paho.golang/paho. autopaho/examples/docker provides a full example using docker to run a publisher and subscriber (connecting to mosquitto).
"fairly simple wrapper" might not be correct any longer; its got a fair amount of functionality now!
Currently I would anticipate that majority of users should be using autopaho; should this not meet their needs then it provides an example that they can copy/modify as required. Note that autopaho is also important to the development of paho because it ensures that paho is usable; it's easy to design a library that looks good but is hard to use (this was very important when implementing session persistence, because a session outlasts a paho.Client instance).
In summary, my feeling is that we already have a solution to the issue you are trying to solve; it's just that we have taken a different approach. We (have discussed this with @alsm) would be reluctant to add reconnection to paho.golang/paho because it's seems like unnecessary complexity. Having paho.golang/paho manage a single connection forces users (including us!) to think about how to deal with things that outlive the connection, and I feel this leads to a better overall solution.
I do really appreciate the feedback and would value your response to the above (we definitely make mistakes and are making a few changes to the paho.golang/paho API to address these!). Personally I've spent way too much time trying to debug the V3 client; this design is a breath of fresh air (even if I've complicated things considerably with the session persistence code!).
Matt
Thanks for the detailed response, @MattBrittan!! ❤️
I must admit, I wasn't aware of autopaho prior to this discussion. It seems like it could be exactly the solution I was looking for. I appreciate the effort put into making paho.golang/paho focused and efficient, and I understand the challenges of balancing complexity and functionality, especially given the history with the V3 client.
I'll take a closer look at autopaho and run some tests to see how well it fits our needs. It sounds like it might be the ideal approach for handling reconnection logic without adding unnecessary complexity to the core library.
Regarding the documentation, I think enhancing the visibility and clarity of information about autopaho could greatly benefit users like myself. Many, I believe, might overlook the brief mention of its auto-reconnect capabilities. A more prominent or detailed section in the README, highlighting autopaho and its features, might be a straightforward yet effective improvement. This could include examples or use cases where autopaho is particularly advantageous.
I'd be more than happy to contribute to enhancing the documentation. Making it clearer that autopaho provides a robust solution for auto-reconnection could guide users towards a more streamlined implementation, especially for those who might otherwise miss it.
I completely agree that maintaining simplicity in the core library while offering extended functionalities through modules like autopaho is a wise approach. Perhaps revisiting the intricacies of connection state management could be an option in the future, but for now, focusing on improving documentation and user awareness seems like a pragmatic step.
Thank you again for the insight, and I look forward to contributing further to this discussion and the project.
@ZekeButlr - always happy to accept contributions (but please do review the guidelines (particularly the note re the Contributor License Agreement). The docs do need work; I've put some time into this recently (including adding an example to the autopaho page) but a lot more is needed.
I'll close this issue off as I've just created a PR that more explicitly pushes new users towards autopaho (part of #215).