ChakraCore Transition
divmain opened this issue · comments
Hello everyone,
In December of 2018, we announced our intent to adopt the Chromium Open Source project in the development of Microsoft Edge. At that time, we made no immediate changes to the ChakraCore project. Now that the new Microsoft Edge is generally available, we believe the time has come to transition ongoing ownership of ChakraCore to the community.
In order to mitigate any disruption, we are committed to providing 12 months of security patches to the most-recent release of ChakraCore (v1.11). This will be true whether or not the community picks up ongoing development. Microsoft will not provide security patches past March 9, 2021. Our hope is that the community will continue supporting ChakraCore after this date.
Some of you may have already heard from us directly, as we have begun to explore community ownership. And, while we haven’t yet reached a quorum of partners to make this viable, we are open and supportive for it to happen.
If you want to participate, or if you have questions or concerns, we want to hear from you.
- Contact us via email at ChakraCoreTransition@microsoft.com
- Contact us on Twitter @ChakraCore
- If you would like to have a one-on-one discussion, please reach out and we’ll set something up.
Lastly, we want to thank the ChakraCore and JavaScript communities for supporting us in our successes and in our failures. We’ve learned a lot and we’ve enjoyed working with you all.
All the best,
The Chakra Team
A note for anyone else thinking about this. Whilst I don't have enough time to take this on on my own I'd be very keen to contribute significantly.
In particular I would be happy to:
- clean up the issue log
- improve the documentation on CC's internal architecture
- fix up the partly completed work to tidy up generators and async functions and enable jitting of them
- Implement some new features including in the short term: top level await and optional chaining
Roughly I'd be able to give CC about 10 focussed hours of input per month + some less focussed time for things like issue triage every day or two.
I've contributed in the past as @penzn, will be interested in working on WebAssembly support and porting - but open to fix other issues.
I seriously doubt that @ppenzin and I will be enough to give CC the level of input it needs - I'm happy to be linked up with anyone else who's thinking of contributing to discuss and work out how we could do it but I think we need 2 or 3 others onboard before we move forward.
I've stated my interest publicly, primarily in hope that it will attract others who are interested but also know they can't take it on alone.
@rhuanjl @divmain we can wait a little before finalizing contributors, but ultimately volunteers would need to get involved, however little their number is. In my opinion any number of volunteers is better than having the project archived - that way there is somebody to keep it up in case more volunteers appear.
I am willing to give it a shot as a contributor, and even think we both should do it, though I agree with @rhuanjl moving project forward with this little people would be a challenge.
What would happen to infrastructure - CI and the like, would Microsoft support it? We don't have to discuss that here if there are better channels.
CC @nmostafa
-
I think there's a few points of clarification on the chakra.dll vs chakracore issue that that will be important to numerous users:
- when you say chakra.dll will be serviced I assume you mean security fixes and the like only?
- whilst chakra.dll is not exactly chakracore it's ~95% the same code therefore to continue adding new features to chakra whilst discontinuing chakracore would be confusing to say the least
- chakracore contains a few sections of code that are excluded at compile time when building chakracore but are included for chakra.dll - a community owned version of chakracore no longer intended for use in chakra.dll would delete those at some point as a matter of housekeeping; whereas if there was ever an intent to update chakra.dll based on the community work those would need to be left in.
-
On the infrastructure question @ppenzin raised - that will be one of several points we'll need to work through if we can find some more volunteers and get this off of the ground. I assume from your response on this point that microsoft is intending to take no official long term role at all? So everything and anything currently done by microsoft in relation to chakracore will need to be transitioned.
Thanks for the clarifying questions.
Yes - chakra.dll will receive security fixes. The scope of this is described in the Windows Security Servicing documentation here. It is not intended that chakra.dll receive feature work, unless (in the very unlikely event) that work is necessary to address a security vulnerability.
If the community takes on continued ownership of the project, any changes (like removing charka.dll-specific code) are entirely up to y'all.
Past March 2021, MSFT does not intend to invest official resources in ChakraCore. That said, if things are moving forward next March and a bit more time is needed to get infrastructure completely transitioned, I think we'll be reasonable and work with you. Certainly interested in making the transition successful.
@divmain What about sources for commit bots - the ones doing merges, CI, various checks, including CLA, would that be released somewhere? This is a substantial part of ChakraCore development workflow.
I'm likely to have some significant unexpected free time over the next couple of weeks so whilst I still think we won't get far without more contributors volunteering.
I could make something of a start perhaps by cleaning up the issue log, closing duplicates and triaging etc. Please could I be given access to do that/what do I need to do to get that sort of access? @divmain
Perhaps if the log looked tidier others may be more likely to volunteer to help.
Greetings, folks.
I am interested in working on ChakraCore as a contributor.
Also, I am interested in participating in community meetings, if they exist.
With 3 of us on board we've got a bit more chance of getting somewhere with this, thanks @Fly-Style
Nothing has happened yet beyond the discussion in this thread and what I've done today making a start at tidying up the issue log - there's still a long way to go there, and I'll continue chipping away at it as and when.
In terms of next steps - here's a quick brain dump of things I think we'll need to sort out
-
We need to think about the process for landing PRs before any more work can actually be submitted, historically any PR has required review by a MS contributor as well as passing all CI before being landed via the commit bot - if the key contributors are just the three of us the review process may be a bit slow + we'll need to know about if/how to use the commit bot.
-
We need to consider how/when this repository will be migrated out of Microsoft land or if we need to begin again with a clone.
-
We need to make a plan for moving the CI out of the Microsoft land
-
We need to consider what the copyright situation is - currently whilst everything is licensed under the MIT license it's also all marked as copyright Microsoft - as Microsoft will not be involved in future contributions do we need to adapt the copyright notices to say Microsoft + future contributors or community team or something?
-
We'll need a plan for handling security issues (assuming there are any major external users) - though we'll also need to consider at the moment that any security issue in chakracore master could also be a security issue in CharakrCore 1.11 and in the latest Chakra.dll as well hence how to collaborate with MS on such.
-
We need a roadmap for feature work and bug fixes/enhancements
-
We need to draw up a release plan/schedule of some kind
-
We need to make this project as attractive as possible for potential users AND potential contributors to get involved
I'm primarily keen to actually write code rather than handle admin points but unfortunately there's no point writing code without some of the above admin.
Thanks for putting the list together. This sounds like a plan!
- We need to make a plan for moving the CI out of the Microsoft land
For the repository, Github has a transfer feature. This would invalidate any paid-for features that might have been used in the old org, but more importantly transitioning the CI and bots can be somewhat involved. @divmain would it be possible to publicly share sources for the bots and CI scripts ChakraCore uses?
- We need a roadmap for feature work and bug fixes/enhancements
I suggest using Github projects to set a few directions of work - for example "JS standard/compliance", "Performance", "Wasm", etc. This would let people to chip away at the issues in any of those directions.
- We need to draw up a release plan/schedule of some kind
In my opinion this can be done in somewhat relaxed, "as needed" manner for now.
- We need to make this project as attractive as possible for potential users AND potential contributors to get involved
One thought is improving CMake build experience, that would make it easier for non-Windows users to contribute. Also, I've meant to looks through Wasm issues, will do that asap. Wasm implementation in Chakra used to be pretty good, though state of "upcoming" Wasm features have slipped.
I'm also interested in working on ChakraCore.
I suggest using Github projects to set a few directions of work - for example "JS standard/compliance", "Performance", "Wasm", etc. This would let people to chip away at the issues in any of those directions.
As a TC39 Invited Expert, I think I can help in "JS standard/compliance" direction and possibly "wasm" direction also.
At the moment, some of JS features are already merged in master (and planned for 1.12), but not landed yet (Promise.allSettled, numeric separators, etc.), so we need to review what is done already. As far as I remember there were some new wasm commits in master also.
@chicoxyzzy Indeed Master is a mile ahead of the last release in features and some performance enhancements - though the issue log indicates that some of those enhancements have brought bugs with them. (There are 1298 commits in master not in release)
Some more thorough documentation of current status would be useful, aside from bugfixes/enhancements I'm aware that master includes the following unshipped features (this is off the top of my head I'm sure that a thorough review would find a few more):
- stable Array.prototype.sort
- object rest and spread
- numeric separators
- Promise.allSettled
- Object.fromEntries
- Array.prototype.flat & Array.prototype.flatMap
- globalThis
- export * as ns
- the /s dotAll option for RegEx
There is also a 95% complete open pull request to finish and enable async generators and async iteration: #6312
Hi everyone! It's really great to see that there's interest in keeping CC going. Although I no longer work at MS (I'm now at Brave) and I don't have a whole lot of free time at the moment, but I'd be happy to help where I can.
I agree with @ppenzin that improving the CMake experience would be a big win.
I also agree that figuring out CI is going to be one of the biggest challenges. Obviously we need to be able to run automated tests against PRs and release branches, but if we'll be touching the JIT at all we'll probably want some kind of automated fuzzing (which might be expensive). I don't think using a personal Azure account is going to work.
@divmain Can you check if there are there any open-source sponsorship possibilities within MS (not necessarily from Edge) that could help out here? It might benefit MS in terms of open-source good-will press, especially if the project becomes self-sufficient and the team does blog posts and whatnot : )
@rhuanjl Perhaps it would make sense to set up a video call to discuss things?
Regarding 4 (MIT license related), that sounds like the type of thing that would require input from a lawyer.
Regarding
4(MIT license related), that sounds like the type of thing that would require input from a lawyer.
I'm hoping it's simple enough that we don't need to do that - there's currently 0 money for this project - and lawyers cost a fortune - additionally the MIT license is designed to be fairly permissive, the only strong stipulation is maintaining the copyright notice. I've definitely seen projects that keep a copyright notice and add another one, I figure that that should be enough - though the easiest option may be if Microsoft who currently own the copyright would like to tell us exactly what to do on this point.
@zenparsing not sure if I'll have time this week for a call, though can definately work something out at some point, maybe next week. Though ideally I would like more people involved in any initial planning meeting - my hope based on the above discussion is that this will very much be a shared project, not me owning it.
On the fuzzing point, is there any chance that google's OSS-Fuzz would be willing to continue fuzzing chakracore? I figure we could ask them - though clearly there's far less reason for it from their perspective than there used to be. (opened an issue on the oss-fuzz repository to ask the question - see link below)
EDIT: OSS-fuzz are willing to provide fuzzing
Picking up on everything above I think we've got a really good chance of getting somewhere with this. A conference call to discuss some of the above points and hammer out an initial plan could be a very good next step.
Would it be possible to have all of us on a call:
@divmain (to answer any questions about support from microsoft)
@zenparsing
@ppenzin
@chicoxyzzy
@Fly-Style
@rhuanjl
I figure that arrangements/timing may be tricky I assume we've got a variety of timezones and other commitments to work around, here's a doodle poll with potential times for this call next week, the options are all of the times I can do, note I'm UK based so timezone is GMT, hopefully we can work something out that will fit most of us:
https://doodle.com/poll/fu33r24zd2mrdcac
I've filed #6392 for CMake.
As for Wasm, @Cellule, @MikeHolman, and @LouisLaf do you have any input or wishes? My goal is to go through Wasm backlog turning notes into issues and triaging open issues.
And lastly, does anybody want to take a look at potential performance optimizations (for standard JS functionality of the engine, rather than Wasm)?
Based on poll responses 7pm GMT today (just over 5 hours from now) is the only option this week - though no reply from @zenparsing or @divmain yet - I'll hold 2 more hours then send out an invitation for that time.
As for Wasm, @Cellule, @MikeHolman, and @LouisLaf do you have any input or wishes? My goal is to go through Wasm backlog turning notes into issues and triaging open issues.
All I can think of is issue #4745 should be updated to latest wasm spec and probably broke down into separate issues for each features
I've opened #6399 to document the developing plans and provide an opportunity for interactive review and comment.
I'm sorry about the short notice and time zone confusion with the call last night - but it was good to discuss with those who could make it - I will try and arrange something with a bit more notice next time.
What will happen to the Chakra-Samples repository? Will it also come under community management?
What will happen to the Chakra-Samples repository? Will it also come under community management?
That repository has been stale for some time - we certainly would like to create a more extensive CC usage sample, though we don’t have a clear plan at the moment. Whether that would be the best place for such a sample is another question.
@zenparsing - I'll look into the OSS sponsorship question. This is an area where I'm significantly investing, building mutually beneficial relationships with OSS projects (not just ChakraCore). So this is certainly in line with my priorities. That said, money is always the hardest ask :) I'll see what I can do!
(Also, hi Kevin!)
As a complete outsider who's never once looked at Chakra before today, and as someone who's been critically assessing the future of the Lua ecosystem as a longtime user of it recently, I really hope Chakra can find a new life as a glue-script. I suppose consider this a rather verbose ++ to what @rhuanjl said in the previous thread on this topic, or perhaps insight from the perspective of a dev that's looking for just the right tool to give users a scripting environment and finds the solutions on offer insufficient.
Basically, right now there are zero ECMAScript JIT engines that are freestanding with C/C++ embedding being their first class citizen. All the extant JITs are from a browser heritage and that shadow looms large.
Lua has been the darling of devs seeking a simple to embed and bind glue-script to surface a readily available API to end users, and has been filling that role in a wonderful fashion for over a decade. But time marches on, things change, and certain design decisions haunt. I could write a very long essay on why Lua is in the state it is, but my point is this: Most people I know still use LuaJit not PUC-Lua, which is stuck at a quasi 5.2 version. Which means in-context strings are treated like binary and pattern matching on Unicode is totally broken.
There is a open niche for a high performance embeddable scripting language engine that lives in the modern era where Unicode is ubiquitous, and Chakra could conquer that niche. Just make embedding a first class citizen. There are tons of embeddables out there, but their design goal tends to be reduced binary footprint, to facilitate support in IoT devices and such. But if I'm building an application that's going to be a few hundred meg on disk anyway, I don't care if the script engine adds another fifteen. I just want something I already know, my users already know, and doesn't have the shortcomings of LuaJIT.
@Xaekai That is exactly the sort of niche I'm hoping chakracore can fill as we continue it as a community project - that said we're talking about continuing it as a team of hobbyists with full time jobs doing other things - so we're unlikely to be always up to date with all Javascript features etc. - but we intend to do what we can and to focus on things that embedders want.
Interesting you mention pattern matching - Chakracore's RegExp engine is probably the part of the engine most in need of improvement - notably it doesn't currently support matching unicode surrogate pairs properly - I was hoping to fix this prior to our first community release but it's more work than I thought and it will likely slip until our second release.