Issues Workflow
chrisdickinson opened this issue · comments
The node and io.js repos approach the issue tracker with different strategies. Node tends towards adding more workflow to reduce duplicated effort across a small group of contributors, while io.js prefers to add more contributors to a smaller workflow. If we're to continue io.js' tradition of adding new contributors regularly – which I am very much in favor of – we need to make sure that the workflow is easy to communicate and internalize. To that end, we should evaluate how the new node tagging process will work with a much larger pool of collaborators.
I'd also like to propose adding a monthly collaborator check-in thread, to keep everyone on the same page on workflow and make sure existing collaborators are comfortable and engaged with the project.
Thoughts on this? Or should this be moved down to a WG which all collaborators are automatically made part of?
To that end, we should evaluate how the new node tagging process will work with a much larger pool of collaborators.
is this defined anywhere?
@Fishrock123 It looks like it went in on node-website.
The new node.js issue workflow items are very new and not yet broadly utilized. It's not clear how well they'll work with a larger more distributed group. The intent is to make it easier to triage issues and it was designed with a smaller group of individuals
in mind.
I think this is basically a dupe of #8, but here's some thoughts on the current node process:
- Milestones Milestones for regular releases (i.e. roughly weekly) are somewhat redundant and unnecessary. Milestones are good for majors (and sometimes minors) where there is more to prepare.
- Priority Labels It may be best to only have either a
criticalorlow-priority, or maybe both. However, I fear this too easily provides extra marginalization for less-considered-"critical" issues. - Status Labels
S-maybe-closeis redundant. Just close the issue. If messaged well, reopening is cheap. - CI for non-doc-fixes: Yes.
My issue with priority labels is that they are predicated on a false assumption: that if we say things are important "to the project" it will end up being important to contributors who will actually write them. Once you have 30+ committers that just doesn't scale, everyone has their own priority list and is knocking it out, all the labels do is make what they are working on seem less important or gives other contributors an excuse not to review things not prioritized.
Chris put it really well when he said that io.js has optimized for a large number of concurrent contributors. In a merged project that would still be the case for current releases, although I do wonder if these kinds of labels might be useful to the LTS WG depending on how many people end up joining it.
A prioritization system works well where limited resources at the bottom of a funnel are tasked with a ton of work making the funnel wide at the top. What we should strive for is a system that isn't funnel shaped but instead has as many contributors and reviewers as the level of work that needs to be done. To do that we need to get out of the way as much as possible and do everything we can to encourage people to do the work important to them and review and merge the work concurrently. The process io.js has isn't perfect, I'm sure we'll continue to iterate and find improvement, but was created and iterated on with these goals in mind where the node.js policies were created to manage an insurmountable workload with limited resources.
@chrisdickinson Right, the priority tags, systematic usage of milestones and assigning issues to collaborators were all suggested and adopted in the context of a very small number of contributors and a large number of issues.
I am wondering though if some of these concepts can be used by working groups or the project at large to execute on the roadmap that has been discussed in both projects. My understanding is that, with a well-defined roadmap, identifying priorities and breaking down large pieces of the roadmap in smaller milestones is useful, if only for communication purposes (e.g to make sure that all collaborators are on the same page).
I think one of the goals of the priorities was to provide some context to new or less frequent contributors when they were choosing issues to work on. They would always be free to work on whatever issue they wanted but if they wanted some input on how to choose it would provide that. I can understand, though that it can make what you are working on seem less important. Many of the ones I've worked on are important to us so that our regular builds pass consistently, but end up being P3 in the overall rating scale. This does not bother me, but that might not be the case for everybody. An alternate might be some sort of +1 by contributors as indication of which ones the overall team would like to move forward.
I think one of the goals of the priorities was to provide some context to new or less frequent contributors when they were choosing issues to work on.
For most of core this just isn't all that practical. Most items on the roadmap are pretty hard and require a decent amount of familiarity with the codebase already, so new users aren't going to have an easy time picking those up. However, it has been brought up multiple times that we should tag issues that are pretty easy low hanging fruit as a target for newcomers but for some reason it hasn't been executed yet.
An alternate might be some sort of +1 by contributors as indication of which ones the overall team would like to move forward.
In io.js the policy is that everything moves forward. If people write a patch it should be reviewed and provided it's a good patch released as soon as practical. When bugs come in we ping the people who can fix those issues and bang the drum on the issue until it gets resolved which is why you'll notice that the "known issues" list in each release is quite small but actually contains all known issues that are logged.
In order to meet this demand we've created contribution policies that allow and encourage people to level up and take on review and triage responsibilities.
I am wondering though if some of these concepts can be used by working groups or the project at large to execute on the roadmap that has been discussed in both projects.
Prioritizing and milestoning what some of the working groups are doing could help a lot. In WGs we find that the work in core is happening concurrently, and in many cases blocking or being blocked by, other projects outside of core. Giving visibility in to not just what you should do now but what you can do would be beneficial. A good example is the Tracing WG who are waiting on a new tracing API in v8 to be integrated as well as integrating AsyncWrap in to userland modules, iterating on internal AsyncWrap, and finally getting better hooks in to v8 in areas like Native Promises. The concurrent release cycles on all these moving parts gets hard to contend with as the group grows. It's not like we wait for it to be possible to use something before there are issues discussing its usage and integration, we should find a good way to signal how close we are to those actually be usable.
Perhaps "priority" is the wrong term for the intended goal then? There are really two things here: 1) prioritizing the things that must get done first (everyone has their own sense of this) and 2) marking the things that may be low hanging fruit for new or infrequent contributors. Rather than using a P-n type tag for the latter, perhaps something along the lines of the novice tag as used in Drupal (https://www.drupal.org/node/1319140, https://www.drupal.org/core-mentoring/novice-tasks). We discussed the notion of a difficulty flag (e.g. D-1) when going over the revised node.js workflow and it was ruled out because what is difficult for some may not be difficult for others, but having something like novice could be a good way to mark those issues that are small and self-contained enough that a reasonably skilled developer could use it as a way of getting their feet wet in the project.
@jasnell 1) I still think prioritizing can be done using objective criteria. I tried to write objective descriptions of all possibles values of the priority label in joyent/node.
- Low hanging fruits are currently marked with the "easy" label in joyent/node. The goal of the priorities label in joyent/node is not to identify low hanging fruits, it is for collaborators to know what issues are on the critical path for a given milestone.
Here's the difference in approach, as best that I can describe it:
Approach A:
- Define a set of "project goals".
- Organize those goals in to milestone releases (I'm assuming these milestones are a set of linear releases)
- Prioritize issues and PRs to achieve each milestone.
- Release as prioritized items get fixed.
Approach B:
- Set a threshold for each new current release (either a number of changes or a set amount of time or a combination of the two)
- Accept and encourage any changeset that bumps the minor or patch after a good review.
- When the threshold is reached do a release.
Approach A has a scaling problem. First you have to get a set of people to agree to a common set of priorities. The more diverse that set of people is the harder that will be and the more confusing the result will be (the compromise here will be a a bunch of different things prioritized so it won't be linear). In the meantime, you're ignoring or at least de-prioritizing the work contributors are doing outside of these priorities which, IMO, de-incentivizes them to stick around and take on more review and triage responsibilities.
Approach B doesn't help the project execute on longer term goals. It doesn't offer contributors who are working together towards a common goal tools to help them achieve that.
Approach B is what we've been doing in io.js and it's really the only way you can manage the number of contributors and TC members we have now. However, if you look through our Roadmap you'll see a bunch of longer term project goals. All of those project goals fall on to various Working Groups and the coordination effort to achieve them is happening there. Those Working Groups are autonomous and prioritize their work however they see fit. This has allowed us to distribute the coordination effort in achieving a bunch of parallel goals and lowered the barrier to entry for contributors to get involved in each of those goals.
In the Working Groups that are trying to achieve some of these goals they will routinely get blocked or hold off on some work waiting for something else to land. This makes a lot of sense because it might be a lot of wasted effort to land something only to have to drastically change it again. The problem you run in to when this policy is applied to core for all or most changesets is that you're constantly shutting down and putting off good work flowing in from contributors. This leads to a stalling of releases and progress on the project and the disengagement of contributors who can't land patches or see those patches land in releases.
The biggest benefit to Approach B as a policy for core changes and current releases is that it emphasizes working code over all else. It tells the community that it is their responsibility to submit changes and it is the project's responsibility to review and ship them, rather than all submissions from community contributors being fed in to project mechanics where they are prioritized for the project to resolve.
It is best to think of the project as not having actual resources to write code at all. If we have priorities it's our responsibility to get the community engaged in fixing them (that's why the io.js roadmap is mostly a slide deck for meetups and conferences and has over 10 localizations). If we think of the project as having dedicated resources to do our prioritized bidding we create a wall between existing contributors and potential contributors, and even throw out some of the existing contributors that don't currently share our prioritization.
I just don't see how the top-down priority approach can work for a project with as many contributors and as diverse a community as a merged io.js/node.js.
I agree that overall people are going to do what they want, and that different groups will have different priorities, particularly as the group gets larger. Given that the value of the priorities set would come into question the benefit could be outweighed by the negative of people thinking their work is being devalued. Is there anything to prevent Working groups (for example LTS, or whatever) from having their own set to aid in their co-ordination as you have outlined above. Maybe something like lts-p1 ? Then working groups could choose to use priorities if appropriate, but we'd not try to prioritize across everything ? Priorities would be domain specific which could reduce the "de-value" side of the equation. Not sure myself if this is better but thought I'd see what others think.
Is there anything to prevent Working groups (for example LTS, or whatever) from having their own set to aid in their co-ordination as you have outlined above.
Nothing should block them, Working Groups are autonomous so they can write and iterate on their own dev policies.
@mdawsonibm they can have them on their own trackers if they prefer. Imo we should only have wg-agenda labels for core or else the labels could begin to become too noisy.
@Fishrock123 working groups already have repos/trackers but they may also need labels in the core issue tracker, which I think is fine so long as you preface the label with the WG name. Those labels don't cost us anything and we can experiment and see what works and what doesn't.
Rather than using a P-n type tag for the latter, perhaps something along the lines of the
novicetag as used in Drupal (https://www.drupal.org/node/1319140, https://www.drupal.org/core-mentoring/novice-tasks).
npm and Node / io.js are operating at different levels of scale, but they share the problem of a large overhang of work and somewhat unapproachable codebases. We've also discussed tagging compact, self-contained tasks with novice, and the biggest challenge there has been coming up with enough of those and then leaving them alone long enough for newcomers to pick up. It's a good idea, but it does require the team to share the goal of making it easy to bring on new developers.
npm has also been following approach B (a regular cadence and fluid goals instead of fixed targets and milestones – the number of contributors is still very small next to either io.js or Node.js), and it is globally slower but locally much more responsive to users with issues. I also find that trying to impose explicit priorities onto a project with a cadence-based release process gets awkward in a hurry; I wouldn't dictate to the working groups, but in my experience labels work much better for categorization and description than they do for prioritization or process management.
S-maybe-closeis redundant. Just close the issue. If messaged well, reopening is cheap.
I agree, except for PRs. Even though it's easy enough to open a new PR if people change their minds, it feels much more aggressive to close a PR unmerged than to close an issue.
Even though it's easy enough to open a new PR if people change their minds
PR's can be reopened too. I think the important bit here is leaving a message along with closing it detailing some of this.
The S-maybe-close was kept in the new "workflow guidelines" document in Node.js because it already existed. I agree with both with @Fishrock123 and @othiym23 that it may be useless, and that at the same time it might be perceived as a bit rough to close an unmerged PR. Anyway this is not the important part of the guidelines, and as stated in the document, it is very optional in our current workflow.
Same thing for the easy label. We tried (and are still trying) to make this work. So far we haven't been successful for the reasons @othiym23 highlighted. It is also optional in our current workflow.
Priorities are also optional and are here to help contributors to focus on what has reached some consensus as being important for the project. It does not prevent anyone from contributing to the project. The io.js project currently moves much faster than Node.js, but the number of issues and PRs is still high enough that I would think it is not trivial for any contributor to know what they should work on in order to maximize their impact on the project. Is it a problem in the io.js project and if so how do you solve that problem?
The "priority" label is actually more of a "severity" label. In that sense it provides guidance on what anyone might want to work on next, but in Node.js if anyone wants to work on something not tagged with a priority, that's totally fine. It also tells how a given issue/PR affects our users, which I think is a valuable information to have around.
As for using milestones, I see it as something that is close to the semver-* tags used in the io.js repository. It is slightly more rigid (it embeds a version number) though and I understand that it could not work for a project that is as fluid as io.js is currently and that uses a very fast release cadence. But for any effort that requires careful progress, with a finite set of tasks and a limited amount of contributors, I still think it can be useful.
For instance, releasing the next LTS release could benefit from having a dedicated milestone that contains all issues/PRs (maybe also tagged with a severity/priority label) that would be considered for that release. That would help the LTS working group identify and agree on what's important for the next LTS release, and communicate that to the rest of the project.
The usage of milestones has not prevented anyone from contributing to the Node.js project so far, as far as I know. It's a very lightweight process. If working groups, especially the LTS working group, are free to adopt that type of workflow if they find it solve some of their problems, not using milestones and/or a priority/severity label for the whole project is totally fine by me.
We also use milestones, but only for majors, and sometimes leading up to a minor.
The io.js project currently moves much faster than Node.js, but the number of issues and PRs is still high enough that I would think it is not trivial for any contributor to know what they should work on in order to maximize their impact on the project. Is it a problem in the io.js project and if so how do you solve that problem?
I haven't seen this manifest as a problem in io.js. The collaborators seem to be mostly self-directing. There are enough collaborators that when they see an issue that is within their grasp, they tend to immediately run with it (vs. tagging and moving on.) Higher priority problems are usually identified by TC members and immediately (implicitly) assigned. New collaborators may still have issues identifying what to work on at first, but most seem to have found a niche to work on pretty quickly – at least anecdotally, I haven't seen newly onboarded collaborators ask for direction on issues to tackle in recent memory.
It also tells how a given issue/PR affects our users, which I think is a valuable information to have around.
The terse "P-*" labelling scheme seems to be at cross purposes to having outside users utilize them. The labelling scheme struck me as oriented at benefitting current collaborators, not end users.
We also use milestones, but only for majors, and sometimes leading up to a minor.
I'm not 100% sure we should continue using them for majors, even :)
My current M.O. around workflow is: does expending this effort enable me to delegate more easily? By delegation I mean: with on this metadata, am I better able to trust that other collaborators are handling the work within a given issue, and focus my efforts elsewhere? Does adding this metadata preclude me from drafting a plan of action elsewhere? Or from asking a collaborator what they're working on? In general, does putting effort into meta-work make my life easier elsewhere? Most label schemes I've seen don't do a great job of that – despite labels and milestones, I still have to read issues and their comments, spend time writing up full proposals for plans of action, and attending meetings to keep abreast of what others are working on.
This might seem a bit selfish, but we're multiplying our workflow out across 30+ people, and most of them are donating their time. We need to respect their donated time, and make sure we're utilizing it the best we can. To that end, any meta-work we ask of them (via labels, merging strategies, milestones, et al) should be of high value and low time investment.
Semver severity is a great label for this reason – these labels save an immense amount of effort when cutting a release. They're low-effort, as well: determining the impact of a given change is already a required part of the review process. I also appreciate tools like the jenkins merge job, or changelog-maker, for similar reasons – the metadata they create is of high value, and the work required to record it is minimal (or automated.)
In a merged project that would still be the case for current releases, although I do wonder if these kinds of labels might be useful to the LTS WG depending on how many people end up joining it.
Besides bug fixes and security patches, what types of issues would LTS contributors prioritize? And how would these prioritize not be aligned with @chrisdickinson's observation that "The collaborators seem to be mostly self-directing.", with the difference being that the collaborators interested in LTS likely being engineers that will self-direct their energy to issues that their LTS sponsor is interested in (Microsoft, IBM, Joyent or some other PaaS).
The way I see it, io.js contributors work on whatever they want to work on. Issues requiring discussion are discussed. Anything that should be back-ported to an LTS release is handled by those with a personal or professional interest in making sure the LTS needs are being met. This is very compatible with the lightweight approach that io.js is already using and @chrisdickinson is promoting. If tasks for the LTS release are not being met, then I see that as something that can be resolved by any one of the companies interested in the LTS release. They merely need to add capable developers to their payroll to work part-time or full-time on the LTS release issues that need resolution. I don't see what responsibility unpaid volunteers have towards patching LTS releases if that doesn't interest them. There is more than enough money in the industry coming from PaaS services to support however many devs are necessary to make sure the LTS releases get all the bug fixes and security patches they need.
However, it has been brought up multiple times that we should tag issues that are pretty easy low hanging fruit as a target for newcomers but for some reason it hasn't been executed yet.
+1
We've also discussed tagging compact, self-contained tasks with novice, and the biggest challenge there has been coming up with enough of those and then leaving them alone long enough for newcomers to pick up. It's a good idea, but it does require the team to share the goal of making it easy to bring on new developers.
Can this not be tackled by having WGs or solo developers working on an epic list novice tasks specific to their larger feature/improvement? The way I see it, novice tasks come in two flavors: (1) the standalone tasks that many here are referring to and (2) smaller tasks that are a small part of some larger effort. In the case of the latter, individual developers or working groups can define any novice subtasks they may have. What would be necessary for this could be a "novice-subtasks" tag applied to any task for which the "owning" WG or developer has taken the time to define such tasks.
The terse "P-*" labelling scheme seems to be at cross purposes to having outside users utilize them.
The labelling scheme struck me as oriented at benefitting current collaborators, not end users.
Yes, the goal for the "P-*" labels is to communicate to collaborators how it affects users, not to have outside users utilize them.
As for being "terse", it uses the P-* naming scheme for various reasons, but most importantly because:
- It allows collaborators to type
Pin the labels text field and immediately see all the values that are available. Using adjectives such as "high", "critical", etc would require collaborators to remember what every priority/severity name is, or look them up somewhere. - Using numbers (such as
P-0,P-1, etc.) indicates a clear relation between different labels that is not subject to interpretation.
As with most things, this naming scheme can probably be improved and your suggestions are welcome and appreciated!
I think the severity/priority label is of "high value and low investment". It allows collaborators not to have to re-investigate/re-read the same issue/PR to determine how it affects users. It doesn't add any work to the normal workflow except clicking on a button to set the label, and it saves others the time of the investigation to determine the severity of the issue.
Anyway, I understand that the larger number of contributors in a merged project could make this useless, and it seems that it hasn't been needed in io.js. I'm not advocating for adding it to the development policy, I am only providing some context and information. If a need for such a label comes up again in the merged project, it's very easy to add it to any workflow.
Anyway, I understand that the larger number of contributors in a merged project could make this useless, and it seems that it hasn't been needed in io.js. I'm not advocating for adding it to the development policy, I am only providing some context and information. If a need for such a label comes up again in the merged project, it's very easy to add it to any workflow.
It sounds like what would make the most sense is to go with the io.js model for a month or so and then circle back around in the TSC to check back in with the node.js contributors who had been using priorities and see what aspects of them they miss in their workflow and use that feedback to continue to evolve the process.
@mikeal 👍