Questions about using tokio
happybeing opened this issue · comments
I'm new to doing stuff in Rust and started my first learning project safe-dash which uses linemux and tui-rs to create a dashboard for monitoring logfiles. I aim to have it gather and update metrics from the logfiles. So far so good!
One thing I'm not sure about is the reliance on tokio, which I understand is an alternative runtime with its own ecosystem of traits, libs etc. I understand you can mix and match this with the 'official' runtime to some extent (not sure of limitations), but with a size overhead (20M?), but I'm not sure!
So I was wondering about the feasibility of a linemux without tokio, why you chose to use it, and the downsides for me (if any) of relying on a lib which uses tokio). This is a much more general question than just about linemux, so I'm not expecting you to answer all this, but any comment would be appreciated. And thank you very much for linemux which has saved me a lot of code and does everything I need.
My project will not be harmed by a 20M overhead, its mainly for learning and so are my questions, although I do want it to be useful both for a specific application, and as a starting point for building dashboards based on logfiles.
Hi, thanks for the interest in linemux!
One thing I'm not sure about is the reliance on tokio, which I understand is an alternative runtime with its own ecosystem of traits, libs etc. I understand you can mix and match this with the 'official' runtime to some extent (not sure of limitations), but with a size overhead (20M?), but I'm not sure!
So the async ecosystem in Rust is somewhat of a can of worms, and I am by no means an authority on the history or some of the decisions made, but have worked with tokio since 0.1 and have seen its mutation over time. I'm not sure about the official runtime you mention, but Rust itself defines the Future trait and async/await operators, which serve as the foundational abstractions for asynchronous design. However, it stops short of defining the framework required to 'execute' these async functions-- thus various libraries have been made to provide executors that manage and drive these Futures to completion. Tokio obviously has its own executor, as well as the futures crate, async-executor, etc. But what other libraries lack is the component which makes doing things like async file and networking I/O possible-- the reactor (which tokio now calls driver). What the reactor does is abstract over the OS-specific polling method (epoll, kqueue, etc.) and integrates those interactions into the Futures paradigm.
But as these components are built, they are built with opinionated design to seamlessly fit a larger model. This has led to some fragmentation in the larger async Rust ecosystem, where deciding your executor means potentially forgoing compatibility with other crates. For instance, tokio uses the common futures-core crate to integrate things like Streams and future composition, but does not use the futures-io crate for things like AsyncRead and AsyncWrite. Instead, it defines its own versions to suit its particular design goals, and provides a separate compat layer since both crates are too popular to ignore the resulting friction. And probably the most driving factor in choosing tokio is the sheer size of its ecosystem-- if another component such as D-Bus or Redis were required, chances are there's a high-quality crate written for it using tokio.
But the size of the resulting library isn't necessarily specific to tokio; rather is mostly an artifact of default compilation settings. There's a repo min-sized-rust dedicated entirely to this issue, and provides various techniques for slimming down resulting binaries. For instance, enabling LTO for release builds brings my lines example down by a factor of ~10:
[profile.release]
lto = true
opt-level = 3$ ls -sh target/debug/examples/lines
17M target/debug/examples/lines
$ ls -sh target/release/examples/lines
1.8M target/release/examples/linesSo I was wondering about the feasibility of a linemux without tokio, why you chose to use it, and the downsides for me (if any) of relying on a lib which uses tokio).
Yeah, it's definitely a cascade of commitment to an ecosystem, much like how the web framework world chooses between angular and react for what their components will be built on top of. Some crates like dbus-rs get around this by splitting out runtime support into a separate crate, but even there tokio is still the only runtime for which support has been implemented. Another option is using feature flags, which I could see actually working for linemux since it's a pretty simple library and doesn't actually expose the tokio-specific parts like tokio::fs::File. I've never actually abstracted over a runtime, so that could be an interesting exercise.
TL;DR: tokio is super-popular, and I just went with what I already knew.
Let me know if I missed something, or anything in general that can improve linemux!
Super response - thank you very much for explaining in such detail. 👏
That's very helpful for me at this early stage. I shall make notes from what you've written and bookmark this to come back to.