Adopt the E2EE cryptography module from matrix-rust-sdk: "Element R"

Question

Adopt the E2EE cryptography module from matrix-rust-sdk: "Element R"

novocaine opened this issue 2 years ago · comments

Context

Element Web currently uses a transpiled version of https://gitlab.matrix.org/matrix-org/olm originally written in C++

Using the rust-sdk's WASM bindings, we replace our entire crypto implementation for performance, stability, security, and maintenance reasons. This project is already approved for work to begin.

The intent is to support both old & new crypto implementations side-by-side through a labs flag up until we're ready to ship.

Update March 2024

Element Web and Element Desktop have been using the Rust crypto stack for new logins for several weeks now. All users on the develop.element.io instance of Element Web have also been migrated to the Rust stack.

Next step, hopefully starting this week (w/c 25 March 2024) is to roll out that migration to app.element.io users.

Update January 2024

We've been continuing to work on defects and missing features in Element Web R.

We've also developed a "migration" process, which will take a session using legacy crypto, and convert it to rust crypto. Once #26772 lands, you will be able to test it out by enabling the "Rust Cryptography" switch in the "labs" settings.

The next stage is to enable it for new logins on "stable" deployments (such as app.element.io, and Element Desktop). The things remaining to be sorted before that can happen are tracked at https://github.com/element-hq/element-web/issues?q=is%3Aissue+is%3Aopen+label%3AZ-Element-R-Blocker.

Update October 2023

https://develop.element.io has been updated so that new logins will use the rust cryptography implementation.

Update June 2023

We are now using a project board to track this work: https://github.com/orgs/vector-im/projects/76/views/15. "Prioritized Backlog" represents the work we consider to be required for the "MVP" of enabling on https://develop.element.io.

To try out the work-in-progress, see matrix-org/matrix-react-sdk#10080.

Plan from December 2022

High level overview of approach

(as of December 2022)

After step 2, Element R web should be good enough to replace libolm-based Element Web as a daily driver. We'll begin to encourage users to try out a hosted instance of it and give feedback, while continuing with step 3 and beyond.

Detailed plans from April 2022

Phases

Phasing is approximate and used as an estimation tool. This list will be updated often - check back frequently.

Time estimates are not guarantees or even accurate - they are relative. They will also be updated as needed.

Phase 1: Prove it works (1-2 months)

[--] Bindings exist on NPM (in progress)
[ 5] Introduce labs flag for js-sdk
[13] Hook up rust stores to storage. Rudimentary one-way migration.
[13] Hook up to sync loop, basic encryption and decryption support.
- Including attachment/media support
[ 8] Key backup (protocol level, not UI)
[ 8] Key sharing (protocol level, not UI)
[13] Cross-signing (protocol level, not UI)
[21] Two-way migration for labs flag, allowing users to "switch" between implementations
- Potential to implement this as an implementation which writes to both stores saving us from migration

Phase 2: Establish trust in the plan (1-2 months)

[ 8] Benchmark old crypto & new crypto to establish performance targets
[??] Tests which prove old crypto was working
- "Working" is defined as behavioural traits, not necessarily bug-free.
[??] Tests which prove the new crypto isn't any more broken than old

Phase 3: Build a plausible client (2-4 months with questionable accuracy)

[??] Self verification
[??] Other user verification
[??] Visual indicators for user trust (cross-signing, 4S)
[??] UI bits for key backup, key sharing, and cross-signing
[??] Manual key export
[??] Historical key sharing (for room history)
[??] Device management

Phase 4: Polish and remaining bits (1-2 months)

[13] Support for customisation endpoints/modules as needed
[13] Matrix Content Scanner integration (if needed)
[??] Device dehydration
[??] Functions required by Element Call and widgets (custom to-device messages?)
[??] Posthog metrics
[??] TBD stuff from https://github.com/matrix-org/matrix-rust-sdk/milestone/1
[??] Get design involved for migration experience

Phase 5: Stability (1-2 months)

[??] Enable by default on Nightly and Develop (not production, EMS, app, or staging)
[??] Fix bugs & build comfort
[??] Enable by default in production (EMS, app, and staging alongside existing develop channels)
[??] Fix bugs & build comfort

Phase 6: Release (1 month)

[??] Convert migration to a one-way migration
[??] Remove old crypto code (keep migrator)
[??] Remove labs flag
[??] Eternal maintenance

Internal references

myhours: https://app.myhours.com/#/projects/1943744/overview

Travis Ralston · Answer 1 · Wed May 18 2022 12:38:38 GMT+0800 (China Standard Time)

waiting on poc to prove this is feasible, then will formulate and enact a plan

James Salter · Answer 2 · Wed May 18 2022 16:02:39 GMT+0800 (China Standard Time)

@Hywan are there some sub issues or PRs that we can link to this epic?

Travis Ralston · Answer 3 · Thu May 19 2022 04:21:37 GMT+0800 (China Standard Time)

matrix-org/matrix-rust-sdk#675 covers the WASM-specific bindings. Crypto team is currently working on a simple encrypt/decrypt demo that will need adapting into production-safe code (which is my problem).

Ivan Enderlin · Answer 4 · Thu May 19 2022 21:54:35 GMT+0800 (China Standard Time)

@novocaine Nothing I'm aware of except the one mentionned by @turt2live hereinabove.

Travis Ralston · Answer 5 · Tue May 31 2022 00:51:56 GMT+0800 (China Standard Time)

Rough tasks that will need breaking out to useful issues (once further narrowed down):

Look at POC
Add labs flag for dual stack crypto
Consider what migration UX might look like: eg element-hq/element-meta#263
Ongoing: perform benchmarks to see if crypto speed is improving (secondary objective)
Get basic encryption working
Get verification working
Get cross-signing/SSSS working
Get backup working (might be related to SSSS these days?)
Figure out what was missed and get it working
Become comfortable with Rust as a crypto engine through unit tests, integration tests, end-to-end tests, and manual/community testing
Remove labs flag, switch to only Rust
Remove legacy code (once sufficient adoption for migration)

Travis Ralston · Answer 6 · Wed Jun 29 2022 05:15:23 GMT+0800 (China Standard Time)

issue description updated with scope

Ivan Enderlin · Answer 7 · Fri Dec 09 2022 15:57:35 GMT+0800 (China Standard Time)

Here is the meta issue for matrix-sdk-crypto-js, matrix-org/matrix-rust-sdk#1016

Richard van der Hoff · Answer 8 · Mon Sep 04 2023 20:54:13 GMT+0800 (China Standard Time)

Update on the current state of play:

We are working towards enabling Rust Crypto on https://develop.element.io. (Once we do this, existing logins will be unaffected, but new logins will use Rust Crypto).

The "Prioritised Backlog" on the project board is an up-to-date view of the remaining work we see as important to complete before that happens (the "MVP") but in summary, the big ticket items are:

#25752: Cross-user verification. This is so close to working we can almost smell it.
#24828: Key backup. Also nearly done.
#25321: I have this largely working locally but need to put up some PRs and improve test coverage.
#24967

Richard van der Hoff · Answer 9 · Tue Oct 03 2023 22:52:21 GMT+0800 (China Standard Time)

#26291 proposes to enable rust crypto on https://develop.element.io

Richard van der Hoff · Answer 10 · Thu Oct 05 2023 22:17:13 GMT+0800 (China Standard Time)

We are tracking remaining blockers for deployment to develop.element.io at https://github.com/vector-im/element-web/issues?q=is%3Aissue+is%3Aopen+label%3AZ-Element-R-Blocker

Richard van der Hoff · Answer 11 · Wed Oct 25 2023 22:02:04 GMT+0800 (China Standard Time)

Just to keep this issue up to date: #26291 has now been merged, meaning that new logins on https://develop.element.io will use rust crypto.

Richard van der Hoff · Answer 12 · Tue Jan 16 2024 01:32:10 GMT+0800 (China Standard Time)

Update:

We've been continuing to work on defects and missing features in Element Web R.

We've also developed a "migration" process, which will take a session using legacy crypto, and convert it to rust crypto. Once #26772 lands, you will be able to test it out by enabling the "Rust Cryptography" switch in the "labs" settings.

The next stage is to enable it for new logins on "stable" deployments (such as app.element.io, and Element Desktop). The things remaining to be sorted before that can happen are tracked at https://github.com/element-hq/element-web/issues?q=is%3Aissue+is%3Aopen+label%3AZ-Element-R-Blocker.

Johannes Marbach · Answer 13 · Mon Mar 25 2024 15:13:43 GMT+0800 (China Standard Time)

The https://github.com/matrix-org/matrix-js-sdk?tab=readme-ov-file#end-to-end-encryption-support section is marked as outdated, pointing at this issue but neither gives any advice on how to set up E2EE using Rust crypto. Is there any recommendation for client developers other than reverse-engineering it from Element Web?

Richard van der Hoff · Answer 14 · Mon Mar 25 2024 18:27:11 GMT+0800 (China Standard Time)

Unfortunately improving the documentation is still on the todo list (matrix-org/matrix-js-sdk#4132).