Rust Support
thomcc opened this issue · comments
So, I was hesitant about filing this, but on twitter you indicated that you weren't opposed to it.
I have a lot of cases where this would be insanely useful, in particular in size-restricted demos, compression code (in particular, often "ad-hoc" compression like that frequently used for unicode tables), and certain kinds of embedded code. Unfortunately, they're largely in Rust, as that's the language I use most these days.
At first I worried that it would be pretty difficult, but taking a look, a lot of this code is pretty general, so maybe it wouldn't be so bad, so I could probably help, and am completely willing to do the work for it (when I have time, although I do at the moment for a bit).
Some things that I think would be needed having looked at the code. Please correct me if I'm wrong, or with stuff I've inevitably missed.
- https://github.com/HBehrens/puncover/blob/master/puncover/gcc_tools.py would need to call Rust/LLVM equivalents (Rust's toolchain is based on LLVM). Or, more likely, a
rust_tools.py/class RustToolswhich implements the same API.- Actually, I'm not sure how much difference is needed here.
nm/objdump/c++filtprobably still work.
- Actually, I'm not sure how much difference is needed here.
- https://github.com/HBehrens/puncover/blob/master/puncover/collector.py#L211-L256 would need a Rust equivalent.
- Some arg parsing changes in https://github.com/HBehrens/puncover/blob/master/puncover/puncover.py, but nothing major really, a
--rustarg, for example.
Does that sound about right to you? Would you be interested in patches for this sort of thing?
Ah, the realization that this tool only supports 32-bit ARM reduces my enthusiasm for doing this a bit, but still, hm.
Thank you for your preliminary investigation! What would help me are examples of both input projects that could aid as a reference as well as tools that would perform some of the analysis manually today. On twitter, you mentioned such a workflow. Could you show that by example?
Also, "size-restricted demos"? Sweet :)
Here are some use cases I'm interested in.
One of my side projects is a regular expression engine, which ends up requiring several unicode tables in order for case conversion, property tests, etc. This isn't open source yet, but you can see an example of case-handling code https://gist.github.com/thomcc/db7ec784d6e14b32695f9d22766c50ca.
Open source examples of similar tables are here: https://github.com/rust-lang/rust/blob/master/library/core/src/unicode/unicode_data.rs, and https://github.com/BurntSushi/ucd-generate, the latter of which I've contributed a bunch to (and the former I probably will in the future, since at least the case conversion tables it has are... 40kb of just data 🙀)
In this space, as you can imagine, you often have tradeoffs where you decrease the size of the table by increasing the complexity and size of the code that reads from the table. It's hard to know if you're making that tradeoff in a good direction.
In the immediate term, this is the use case I care about the most, but solely because it's a current side-project of mine (mostly because it's a very fun problem space IMO — balancing decode performance, code size, and data size... especially given that general purpose compression isn't a great fit, given that the tables are pretty small to begin with, and need random access).
Also, in the past, when I worked on Firefox, I was also quite concerned with the code size of the rust code my team was responsible for, and had to trim it down many times for mobile. Then I used tools like https://github.com/google/bloaty, or the rust-specific ones like https://github.com/RazrFalcon/cargo-bloat, and https://github.com/rustwasm/twiggy.
These often will tell you a function is too big, but not why it's too big, not even including asm, and often ignoring or mostly-ignoring static data size. They also don't do as good of a job revealing information in the UI as puncover, and don't provide much that puncover can't (afaict) figure out already.
I think puncover, from what I can tell, gives an easier to comprehend view of this stuff than anything I've seen, and lets you drill down into the details of why something is too large — like taking a look at its assembly. None of the tools I just linked offer that, and last I checked nor do they even tell you information about static data used by the function — it's all the same to them.
So, you end up using nm, objdump, etc. Which AFAICT is more or less what puncover does. So, I think to a large extent, the work puncover needs here would be somewhat small.
Other popular use cases for Rust where code size matters (but not for me, at least at the moment) include wasm stuff, since targetting it is so easy (much easier than wrangling emscripten). This is popular both for web stuff (where size matters because you don't want to have the user download too much stuff), as well as for running code on cloud compute providers (where there's often a hard limit on code size — cloudflare workers, which I used to work on, had one of 1MB).
That said, personally I'm not that invested in this area. Additionally, wasm is such a weird binary format that it needs special tools. Not to mention, if you reduce size on one platform, generally that's strongly correlated with reducing size on another.
Also, "size-restricted demos"? Sweet :)
Well, it's been a while (although now that I'm back in games I'm feeling the itch again...) However, for these you often also use exe packers/compressors, which... complicate things... And are probably reasonable to declare out of scope
Sorry for disjointedness, my caffeine is still kicking in.
Hopefully this makes sense and answers your questions, the TLDR for tools I use manually is: either nm/objdump, or... honestly also just compiling a small library, and directly measuring the size (like, with wc -c), coupled with compare two versions of the code that way (e.g. normal stuff you do when tooling is bad).