Modularize and decouple from rust-windowing

Question

Modularize and decouple from rust-windowing

rib opened this issue 2 years ago · comments

Robert Bragg commented 2 years ago

Following up from #365 here is a specific plan for us to modularize this repo

Why?

Several of the crates are very orthogonal from each other:

ndk-glue in particular is unlikely to see continued development / maintenance so it would make sense to clearly sign post from here to android-activity and provide an archive for posterity
cargo apk may eventually be superseded by something like xbuild and it's also not inherently coupled to the the other ndk components
Eventually we may aim to make ndk-context redundant if we can instead encourage crates that support Android to have explicit initialization APIs that may be given a jvm + Context/Activity (and therefore enabling multiple Activities in a single process)
None of these crates are inherently rust-windowing specific and there is probably more opportunity to foster collaboration around an organisation that's more broadly related to mobile OSs in general.

Where to?

Since it's anticipated that most people interested in Rust + Android are likely also interested cross-platform mobile support, we have created a general rust-mobile organization, initially owned by the contributors to android-ndk-rs and android-activity.

The idea is that I will also move android-activity here, along with splitting the android-activity/examples out into a new repo.

I believe @dvc94ch was also open to moving xbuild here too

Other Rust utilities and build tools etc catering to supporting Rust development for mobile operating systems would also be welcome to create repos here

How?

These are key repos that would be split out of android-ndk-rs:

ndk-glue - A filter of the ndk-glue and ndk-macro subdirectories, updated with a notice to recommend android-activity for new projects and then archived for posterity.
cargo-apk - would be a filter of the cargo-apk + ndk-build subdirectories and ndk-examples (renamed to `examples) and outstanding PRs + issues would be transferred to the new repo
ndk-context - A filter of the ndk-context subdirectory
ndk - After removing all the subdirectories that have been filtered into standalone repos this repo would then be transferred to rust-mobile (so all issues / PRs are preserved)

How to filter the repos:

git clone https://github.com/newren/git-filter-repo

ndk-glue

git clone https://github.com/rust-windowing/android-ndk-rs.git ndk-glue
cd ndk-glue
py ../git-filter-repo/git-filter-repo --path ndk-glue --path ndk-macro --path ndk-examples --path-rename ndk-examples:examples --path LICENSE-APACHE --path LICENSE-MIT --path Cargo.toml --path rustfmt.toml --path .github --path .gitignore

cargo apk

git clone https://github.com/rust-windowing/android-ndk-rs.git cargo-apk
cd cargo-apk
py ../git-filter-repo/git-filter-repo --path cargo-apk --path ndk-build --path ndk-examples --path-rename ndk-examples:examples --path LICENSE-APACHE --path LICENSE-MIT --path Cargo.toml --path rustfmt.toml --path .github --path .gitignore

ndk-context

git clone https://github.com/rust-windowing/android-ndk-rs.git ndk-context
cd ndk-context
py ../git-filter-repo/git-filter-repo --path ndk-context --path ndk-examples --path-rename ndk-examples:examples --path LICENSE-APACHE --path LICENSE-MIT --path Cargo.toml --path rustfmt.toml --path .github --path .gitignore

Coordination

What needs to get done?

ndk-glue

Filter out the ndk-glue repo and push to rust-mobile
Get CI working for the standalone ndk-glue repo
Update repository + homepage URLs in Cargo.toml for the standalone ndk-glue repo
Update the README for ndk-glue to recommend that new projects use android-activity
~~Open + merge a PR to remove the ndk-glue and ndk-macro subdirectories from this repo~~

cargo-apk

Filter out the cargo-apk repo and push to rust-mobile
Get CI working for the standalone cargo-apk repo (including porting the examples to build against android-activity instead of ndk-glue)
Update repository + homepage URLs in Cargo.toml for the standalone cargo-apk repo
~~Open + merge a PR to remove the cargo-apk, ndk-build and ndk-examples subdirectories from this repo~~

ndk-context

Filter out the ndk-context repo and push to rust-mobile
Get CI working for the standalone ndk-context repo
Update repository + homepage URLs in Cargo.toml for the standalone ndk-context repo
~~Open + merge a PR to remove the ndk-context subdirectory from this repo~~

removals from this repo

Open + merge a PR to remove ndk-glue, ndk-macros, cargo-apk, ndk-build, ndk-examples and ndk-context from this repo

This PR will indicate:

Where the code is moving too
How the repos were filtered
The last commit used as input for filtering

As follow up tasks we will:

Mark the ndk-glue repo as "archived" on GitHub for posterity
Transfer outstanding cargo-apk issues / PRs from this repo to the standalone repo
Transfer this repo to rust-mobile
Update repository + homepage URLs in Cargo.toml for this repo
Update the README for this repo to reflect the modularization changes

I'm happy to get the ball rolling with re-filtering the repos (since we already did one draft run of this a while ago) as well as getting CI bootstrapped for the standalone repos (I already have a draft for getting the cargo apk CI working and porting the ndk-examples to build against android-activity)

Does this plan sound practical?

David Craven · Answer 1 · Tue Dec 06 2022 23:33:48 GMT+0800 (China Standard Time)

Sorry for dropping the ball on this. We have merged a few PR's to xbuild recently that have significantly closed the gap in terms of features supported by xbuild/cargo-apk.

I'm still happy to transfer the xbuild repo to rust-mobile, just haven't pulled the trigger on it yet.

The remaining open question for me is who has access to this repo and authorization to transfer it to a different org?

David Craven · Answer 2 · Tue Dec 06 2022 23:36:12 GMT+0800 (China Standard Time)

ok pulled the trigger on my end xbuild is moved

Robert Bragg · Answer 3 · Tue Dec 06 2022 23:37:13 GMT+0800 (China Standard Time)

Someone with write access to this repo would also need to handle the transferring of any PRs/issues for cargo-apk that should be preserved, ref: https://docs.github.com/en/issues/tracking-your-work-with-issues/transferring-an-issue-to-another-repository

Robert Bragg · Answer 4 · Tue Dec 06 2022 23:40:04 GMT+0800 (China Standard Time)

Just looking at re-filtering ndk-glue, I've tweaked how I filter it out so that it also includes the hello_world.rs example out of ndk-examples which can be used to get the CI green before the repo would be archived.

Robert Bragg · Answer 5 · Wed Dec 07 2022 01:48:19 GMT+0800 (China Standard Time)

ok pulled the trigger on my end xbuild is moved

I've just moved the android-activity crate over to the rust-mobile organization too

Robert Bragg · Answer 6 · Wed Dec 07 2022 02:44:57 GMT+0800 (China Standard Time)

Okey, I think the standalone ndk-glue repo is in a decent shape at this point. The CI is all green: https://github.com/rust-mobile/ndk-glue/actions/runs/3632277679 and the READMEs have been updated to note that the project is no longer actively maintained.

I think it'll also make sense for the ndk-context crate to filter the ndk-examples to keep the examples/hello_world.rs and that can be tweaked to make some minimal use of ndk-context + jni-rs to smoke test that ndk-context works. This way the existing CI that builds and runs the hello_world example can continue to be leveraged.

Robert Bragg · Answer 7 · Wed Dec 07 2022 07:48:40 GMT+0800 (China Standard Time)

Since looking at ndk-context I've also realized it can't use --subdirectory-filter ndk-context since the examples need to continue being maintained under a separate package.

The examples package needs to use a [patch] to make sure android-activity uses the same (local) version of ndk-context - otherwise it would be initializing global state associated with the "0.1.1" release of ndk-context that's different to what the examples will try and refer to.

Robert Bragg · Answer 8 · Wed Dec 07 2022 09:25:54 GMT+0800 (China Standard Time)

As a bonus (since CI didn't actually have any host tests to run for ndk-context previously) the standalone repo has a patch to get the Doc-tests working before updating the CI for the standalone repo.

Robert Bragg · Answer 9 · Wed Dec 07 2022 09:43:48 GMT+0800 (China Standard Time)

I think it could be simpler to just have a single patch / PR that removes everything that's been split out in one go instead of dealing with order dependent patches and ensuring the CI is updated for each intermediate stage of removal. In case it's necessary to iterate I think that'll be easier to handle too.

Marijn Suijten · Answer 10 · Thu Dec 08 2022 18:48:01 GMT+0800 (China Standard Time)

ok pulled the trigger on my end xbuild is moved

Should we also move cargo-subcommand?

For the rest, I am under enough time pressure lately that I cannot really look into all this separation/cleanup for now, thanks for understanding.

David Craven · Answer 11 · Thu Dec 08 2022 19:11:48 GMT+0800 (China Standard Time)

Should we also move cargo-subcommand?

done

For the rest, I am under enough time pressure lately that I cannot really look into all this separation/cleanup for now, thanks for understanding.

nothing particularly difficult here. @rib is splitting up the repos as discussed, I can approve the PR. the main thing is we don't merge any PR's in this repo anymore that aren't to the ndk and ndk-sys crates.

Robert Bragg · Answer 12 · Thu Dec 08 2022 20:30:37 GMT+0800 (China Standard Time)

For the rest, I am under enough time pressure lately that I cannot really look into all this separation/cleanup for now, thanks for understanding.

No worries, I think I am nearly done with the bulk of the changes to modularize the repo. The last thing was just to make the patch that removes cargo apk from this repo.

I'll hopefully get that ready soon.

I think the filtered repos on rust-mobile are in good shape already but it could be good if you can at least cast an eye over them (mainly the cargo-apk repo).

As @dvc94ch said, it would be good to avoid landing changes outside of ndk and ndk-sys to avoid having to redo the filtering ideally.

Marijn Suijten · Answer 13 · Thu Dec 08 2022 20:58:07 GMT+0800 (China Standard Time)

The main point is also that I need to touch xbuild and cargo-apk for that every once in a while (and I have a release imminent after fixing up --manifest-path and the recent #375 regression), so I just hope we're not stepping on each others toes there.

Especially that new rule of only using this repo for ndk and ndk-sys may become problematic for that, unless I start setting up and using the new repos (and get some help migrating the repos).

Perhaps we should pull in @rust-windowing/publishers (I can do it myself, just looking for some acks) to move this entire repo as a whole to rust-mobile and rename it to ndk.

Robert Bragg · Answer 14 · Thu Dec 08 2022 21:08:22 GMT+0800 (China Standard Time)

I think the cargo-apk repo itself on rust-mobile could be good to go, though we should probably wait until merging the PR as the final commitment for when to switch (just incase we want to make a last moment tweak to it). One minor tweak I was considering was just to update the commit message that fixes CI for the standalone repo to just repeat the information about how the repo was originally filtered from this repo.

Robert Bragg · Answer 15 · Thu Dec 08 2022 21:44:29 GMT+0800 (China Standard Time)

And yeah some additional ACKs e.g from @msiglreith and perhaps earlier contributors like @m64 and @francesca64 could be good to get here.

Mads Marquart · Answer 16 · Thu Dec 08 2022 22:41:12 GMT+0800 (China Standard Time)

Thanks for the ping - I don't have a problem with moving this away from @rust-windowing, if you think it'll help you find more maintainers and such.

I second the need for an ACK from @msiglreith, since they maintain the Android backend for winit and will be affected by this - it'll probably make sense to add them to the @rust-mobile org as well.

Wrt. crates.io ownership:

cargo-apk and android_glue is owned by @tomaka
ndk-sys, ndk-glue, ndk-build and ndk is owned by @Osspial
ndk-context and ndk-macro is owned by @dvc94ch

While @rust-windowing/publishers can publish and yank to those crates, we can't change their ownership - and as such I can't help you transfer those to something like @rust-mobile/publishers (I'm willing to add others to @rust-windowing/publishers if need be though, just say so if you'd like to).

Robert Bragg · Answer 17 · Thu Dec 08 2022 23:43:33 GMT+0800 (China Standard Time)

I guess it would make sense for @MarijnS95 to get ownership rights on crates.io for the cargo-apk crate from @tomaka.

Markus Siglreithmaier · Answer 18 · Thu Dec 08 2022 23:48:30 GMT+0800 (China Standard Time)

I'm fine with the transfer, more collaboration is welcome (currently can't help much myself tho)

Robert Bragg · Answer 19 · Thu Dec 08 2022 23:48:42 GMT+0800 (China Standard Time)

.. oh and nkd-build ownership from @Osspial (since the standalone repo for cargo-apk also contains ndk-build)

It sounds surprising that rust-windowing/Publishers cant approve additional crate owners, are you sure that's the case? @madsmtm

Robert Bragg · Answer 20 · Thu Dec 08 2022 23:58:43 GMT+0800 (China Standard Time)

I second the need for an ACK from @msiglreith, since they maintain the Android backend for winit and will be affected by this - it'll probably make sense to add them to the @rust-mobile org as well.

btw just to note that, yep, @msiglreith is already a co-owner of the rust-mobile organization 👍

Markus Siglreithmaier · Answer 21 · Fri Dec 09 2022 00:00:08 GMT+0800 (China Standard Time)

Regarding team owners, it doesn't seem to be possible : https://doc.rust-lang.org/cargo/reference/publishing.html

If a team name is given to --add, that team is invited as a “team” owner, with restricted right to the crate. While they have permission to publish or yank versions of the crate, they do not have the ability to add or remove owners.

Robert Bragg · Answer 22 · Fri Dec 09 2022 00:08:12 GMT+0800 (China Standard Time)

ah okey, interesting.

I'd figure that to start with the main ownership change that would make sense to me is that @MarijnS95 would get ownership for cargo-apk and ndk-build since the new standalone cargo-apk repo won't be associated with CI permissions for publishing these crates.

It could also make sense for @MarijnS95 and @dvc94ch to get ownership of the ndk-glue and ndk-macro crates but since those are being deprecated it's probably not too much of an issue.

Having ndk-context be owned by @dvc94ch sounds fine

I guess it could make sense for more people including @MarijnS95 to co-own the ndk + ndk-sys crates but since these are staying in the existing repo I expect the existing permissions associated with CI hopefully stay in tact even if the repo is transferred between organisations.

Robert Bragg · Answer 23 · Sat Dec 10 2022 02:36:43 GMT+0800 (China Standard Time)

Ok, I think we might be in a position where we can commit to the modularization via #374 now (I recently pushed a patch for the removal of cargo-apk, ndk-build and ndk-examples).

@dvc94ch has already given it a thumbs up and I think it'd be good if @MarijnS95 can also take a look too if possible.

The other checklist items in the plan above can be tackled in follow up changes after the initial modularization so then we can minimize the time that we pause landing changes in this repo.

Dealing with updating the crates.io ownership permissions can also hopefully be dealt with asynchronously - though it would be nice to get a ACK from @tomaka about adding @MarijnS95 as an owner for cargo apk and an ACK from @Osspial about adding @MarijnS95 as an owner for ndk-build.

Mads Marquart · Answer 24 · Sat Dec 10 2022 09:23:23 GMT+0800 (China Standard Time)

I doubt you'll get them with a GitHub ping, but it would probably make sense to send them an email about it at some point

Robert Bragg · Answer 25 · Mon Dec 12 2022 04:56:03 GMT+0800 (China Standard Time)

As noted here #376 (comment) I hadn't fully appreciated that @MarijnS95 is also a member of rust-windowing/Publishers currently which means there's no short-term issue with committing to use the standalone repos because @MarijnS95 would still be able to publish releases from the new repo.

I was thinking there were credentials associated with the CI for this repo that @MarijnS95 relied on to publish releases.

It would probably still make sense for @MarijnS95 to be a co-owner for cargo-apk and ndk-build on crates.io but it's not a blocker for switching to the standalone repos.

David Craven · Answer 26 · Mon Dec 12 2022 05:41:12 GMT+0800 (China Standard Time)

@madsmtm can you transfer the repo to rust-mobile? that would simultaneously solve the problem of @rib not having write access.

David Craven · Answer 27 · Mon Dec 12 2022 05:44:47 GMT+0800 (China Standard Time)

just tried transfering an issue. I think you can only transfer an issue within the same organization

Robert Bragg · Answer 28 · Mon Dec 12 2022 06:12:04 GMT+0800 (China Standard Time)

just tried transfering an issue. I think you can only transfer an issue within the same organization

okey, yeah, sounds plausible. At least for the two PRs that were recently opened for cargo-apk I was able to make equivalent branches based on the original patches and have opened new PRs against the standalone repo.

Marijn Suijten · Answer 29 · Wed Dec 14 2022 17:01:40 GMT+0800 (China Standard Time)

Someone with write access to this repo would also need to handle the transferring of any PRs/issues for cargo-apk that should be preserved, ref: https://docs.github.com/en/issues/tracking-your-work-with-issues/transferring-an-issue-to-another-repository

Ran into this last week that I cannot transfer issues outside of the rust-windowing orga (to the new repos on rust-mobile), but that hopefully resolves itself when this repo is first transferred there 🤞.

I guess it could make sense for more people including @MarijnS95 to co-own the ndk + ndk-sys crates but since these are staying in the existing repo I expect the existing permissions associated with CI hopefully stay in tact even if the repo is transferred between organisations.

For completeness, as discussed in #376 the publish rights will stay intact as long as I remain part of @rust-windowing/publishers, at least until crate ownership is granted. I already received the rights from @tomaka for cargo-apk (thanks!) and now we're waiting for the rest. The CI uses a personal access token since it is not possible to make one for an organization, but it might make sense to create a rust-mobile CI "user" account that only has access to publish these specific crates. Again, as you mentioned above, that first requires access to the crates unless we add this user to @rust-windowing/publishers for the time being which is weird.
(Edit: looks like some of this was already covered in followup conversations, but most should still be relevant)

@dvc94ch has already given it a thumbs up and I think it'd be good if @MarijnS95 can also take a look too if possible.

Agreed. I think the split here is done and ready to be merged, would still like to go over the other repos through for a sanity check (mostly did that, you've seen the commit reviews by now) and hope we can add/update the readme's sooner than later - especially to make new visitors aware of this split.

All that remains then is to rename this repo and transfer it to rust-mobile, @madsmtm?

Mads Marquart · Answer 30 · Thu Dec 15 2022 01:33:29 GMT+0800 (China Standard Time)

I already received the rights from @tomaka for cargo-apk (thanks!)

Cool! Just a suggestion, I'd recommend adding someone else to that from the org, so as to reduce the bus factor ;)

All that remains then is to rename this repo and transfer it to rust-mobile, @madsmtm?

I'll try, sorry for the delay

Mads Marquart · Answer 31 · Thu Dec 15 2022 01:35:48 GMT+0800 (China Standard Time)

I think you need to give me permission to create repositories in @rust-mobile (just temporarily) - otherwise, @msiglreith can probably do it (they have admin rights in @rust-windowing as well)

Marijn Suijten · Answer 32 · Thu Dec 15 2022 17:08:31 GMT+0800 (China Standard Time)

@madsmtm Whoops, I invited you with admin permissions (needed for the transfer, apparently)!

Marijn Suijten · Answer 33 · Thu Dec 15 2022 18:45:03 GMT+0800 (China Standard Time)

Nice, looks like you also took care of renaming it to ndk 🎉

Mads Marquart · Answer 34 · Thu Dec 15 2022 18:45:26 GMT+0800 (China Standard Time)

Yeah, I've completed the transfer to https://github.com/rust-mobile/ndk and removed myself from the organization again ;)

Robert Bragg · Answer 35 · Fri Dec 16 2022 06:53:34 GMT+0800 (China Standard Time)

Cool, thanks @madsmtm

David Craven · Answer 36 · Thu Dec 22 2022 21:58:28 GMT+0800 (China Standard Time)

so think we can close this one too

Marijn Suijten · Answer 37 · Thu Dec 22 2022 22:10:50 GMT+0800 (China Standard Time)

Yeah, I think @rib is only using this issue to track the URL updates which I already happened to tackle in #378 (because the documentation = was already wrong...), and the README updates across repos which still need to be submitted.

Robert Bragg · Answer 38 · Thu Dec 22 2022 22:28:11 GMT+0800 (China Standard Time)

Okey, yeah no worries. I had been intending to continue using this issue to track follow up work for the modularization but I'm sure we'll figure it out ok from here.