SC Sign off on OSPO Survey 2020
caniszczyk opened this issue · comments
We are making final modifications to the OSPO survey and would like to get the SC to sign off :)
Hey there!
On today's SC call we decided that the best approach would be for @caniszczyk and @LawrenceHecht to finish their flurry of changes and, once the survey is no longer a moving target, for one of them to create a PR that the SC can review. That PR would then become the final, approved version of the survey.
Here is the pull request for people to review. #63. @caniszczyk or @vmbrasseur how do we go about getting sign off?
OK, survey questions are here for review: https://github.com/todogroup/survey/blob/master/2020/questions.md
Question 9 needs an answer which includes both formal and informal; a number of us have both formal (assigned roles) and informal (volunteers, champions, etc.) in our overall programs. It's very common.
Question 13's answer '0 (no dedicated staff yet)' conflicts with answers of a purely informal program; orgs with informal programs should supply a non-zero answer even though the staff is not dedicated.
Question 21 should offer the same additional answer as requested for Question 9.
How are companies selected for question 32's answers?
@kpfleming, here are some responses:
Question 9
- The purpose of the question is to determine if there is a formal program. Instead of rewriting the question, we can tweak the wording so it reads, "How is the primary program or initiative at organization structured?"
- An alternative approach is change the question to "Is open source program or initiative formally organized with dedicated person-hours, reporting structure and/or job titles?" {Yes, No, Don't know}
- Then, we would change make Question 13 be the next question asked.
Question 13
- I think this question should focus just on people with formal roles or working. If so, then I will add logic so that it is only asked of people with formal roles.
- Otherwise, the question should be asked in such a way so that we learn about how many people are involved formally and then ask how many more people are involved informally. I don't favor this approach because some people will think that anyone in a Slack group or mailing group should be counted.
###Question 21 - Yes, whatever happens for Question 9 should happen here.
###Question 32
-Here is text explaining how they were picked: "We asked respondents this question about 11 TODO Group members, which represent a broad cross-section of technology companies. All TODO Group members have a significant investment in open source and many have dedicated open source programs to set policies and encourage contributions."
I like the second option for Question 9, and agree then that Question 13 would follow on only if Question 9 is answered "yes". I do think there could be some value in asking how many additional people are involved in informal roles, but it will be hard to ask the question in such a way to gather only people who have a responsibility in the program, and aren't just people who hang out in chats/lists. In my case I have ~10 people who are volunteers but who have primary responsibility for reviewing outbound contribution requests.
For question 32 - companies included; in prior surveys they were selected from the TODO group member companies, with an attempt to select a "like" industry cohort - in this case "tech" companies. Is this true again this time around? If Oracle is included, are they a TODO group member? We need a defensible selection criterion
@SuzanneA300 correct -- we took Oracle out of the list because we couldn't easily come up with a defensible criterion.
Ah - I see that Uber is now substituted. Thanks.
Question 43: suggest that you add the LF ACT Initiative (automated compliance tooling)
Question 46: add innovation speed as a benefit ?
@SuzanneA300 for Question 43, this is how we can update the choices:
- Delete FOSSology and SPDX
- Add "Automated Compliance Tooling Initiative: projects include Software Package Data Exchange (SPDX), Tern, OSS Review Toolkit, FOSSology and Quartermaster"
Regarding question 32: there are a lot of TODO Group members who are 'tech' companies, if that is defined by the current list. I would include at least these:
- Adobe
- ARM
- Autodesk
- Baidu
- Box
- Datadog
- DiDi
- Dropbox
- Dell/EMC
- GitHub
- Here
- HP Enterprise
- Huawei
- Indeed
- Juniper Networks
- National Instruments
- Netflix
- PayPal
- Pivotal
- Qualcomm
- Sauce Labs
- Spotify
- Square
- Stripe
- Tencent
- Verizon Media
- Wipro
As you can see, it's going to be hard to have a reasonably-sized list and also have defensible criteria for who is included in the list; even if you tried to limit the list to 'large' companies (by market value or number of employees) you'd only remove a few.
Question 5 - Recommend changing the phrase "spare time" to "personal time." No one has spare time.
Question 7 - I understand why we are asking this, but I'd like to see us iterate on the language, to frame it as more of a positive and to remove the subtle time-bound nature of the question (by the time the survey is released, I expect most companies will already have done some kind of reevaluation, which makes this sort of a future-tense question about past events). Recommendation below:
- In light of recent macroeconomic conditions, what is the likelihood that your company will maintain its expected 2020 investments in open source initiatives?
@kpfleming 1) The phrase "represent a broad cross-section of large technology companies" seems defensible. Perhaps if Tencent were added, then that would represent a large Chinese non-tech.
Question 9 - How do folks feel about framing this by asking them to choose a description?
"How would you describe the structure of your open source program/initiative?"
- We have a formal program with full time employees who work only on open source
- We have an informal program which is largely driven by internal volunteer efforts
- We have a formal program with a blend of dedicated staff and internal volunteers
Question 32 - As a data point, of the 11 companies in that list of TODO group members, 1-3 of them are not what I would describe as active participants.
+1 to the call for a defensible selection criteria. A consideration here is that when measuring things like reach and share of voice, there is some credence put on proximity mentions (being mentioned in the same sentence as your peers). I don't have a good recommendation here, but I wonder what results we would see if we asked this as a two part, free form question
"What companies would you consider to be good open source community members?"
"What companies would you consider to be bad open source community members?'
In previous years I found the company list for question 32 to be non-inclusive and not understandable, and I don't think it's gotten better. The general population doesn't consider Comcast to be a 'tech' company any more than they consider Bloomberg to be a 'tech' company. The list of TODO Group members who are also 'large technology companies' is much longer than what is included in the survey, and at least some of the missing companies have contributed as much (or more) to the open source community as companies who are on the list. There are also a number of companies who should be on this list but are not TODO Group members (HashiCorp comes to mind, I'm sure I could easily find 6 more without much effort). When I've filled out this survey in the past, the list of companies read as if it was a 'sponsor list' because it was so short; I know it's not, the sponsors of this survey are well indicated, but I've completed enough analyst/research surveys in the past to get that impression from this question.
In fact only one of the companies represented by the current TODO Group Steering Committee are in this list (AWS). The remainder are GitHub, Bloomberg, Indeed, Spotify and Juniper Networks.
In any case, at a minimum VMware needs to be changed to Pivotal / VMware, just like IBM / Red Hat.
Question 36 - It seems weird to have both the LF and an LF Umbrella Foundation in this list
Question 39 - I would LOVE to see this paired with "How large is your company's engineering organization" and "How many employees does your company have"
Also recommend asking how many employees contribute open source, rather than focusing only on developers. Season of Docs would be an argument for broadening the language here.
Questions 43 and 44 - Can you clearly articulate the difference here between a Methodology/Initiative" and a Tool? ClearlyDefined isn't so clearly one or the other (ironically)
Question 44 - Given the reach of some of these tools, I question Tidelift's absence from this list.
WRT to changing the VMware entry to Pivotal/VMware -- in this instance it should be ONLY VMware. With the acquisition the Pivotal brand is no more. In the case of Red Hat and IBM -- the Red Hat brand was retained and in essence forms a pseudo-subsidiary. Pivotal is fully integrated. So, please leave VMware as is.
WRT to company selection in Q32 - does anyone have suggested criteria for inclusion or exclusion? TODO Group membership... LF Member (Silver or above)... company size... industry... searching for a valid set of criteria that we'd feel comfortable with and comfortable defending. But also don't want to have a LONG list of names for respondents to wade through...
Can we attack that from another angle? What is anyone going to do with the answers from question 32? This isn't a brand reputation survey for the companies in question, so looking at the broader open source ecosystem I can't figure out how the answers to that question are valuable. If we know the answer to that question we can come up with a list of candidate companies more easily.
As a reminder, we used Q32 last time for this article: https://thenewstack.io/survey-shows-how-developers-and-their-employers-measure-good-open-source-citizenship/.
Others have tried a different approach. See how I wrote up Digital Ocean's attempt: https://thenewstack.io/the-value-of-big-tech-in-open-source-sustainability/.
Thanks, I had forgotten about that. Given that, I see that this is solely a brand-reputation exercise, and there's no practical way to extend it to the long list of companies who deserve to be on the list. If this year's results show Salesforce getting a better rating, for example, that's not actionable in any particular way, it's just good news for Salesforce.
@DuaneOBrien re: Q43 and Q44 -- we were not trying to ask about specific tools.
For Q44, I was only trying to include the major software composition analysis players. I don't think Tidelift fits into this mix, but the definition of this market is crazy. If we add any more companies (and I don't really want to, I think these would be the two at the top of my list:
@nruff, Right now we are asking about 11 companies. For me it is not acceptable to include more than 14 in the list. Here is my suggestion for the companies to include:
AWS
Microsoft
Google
Facebook
Tencent
SAP
Salesforce
IBM
VMware
Uber
Netflix
Comcast
Verizon
And, I suggest adding "The following list represents a range of large companies that use open source." to the front of Q32, so it would now read "The following list represents a range of large companies that use open source. To what degree do you perceive each of them to be “good open source community citizens” in terms of contributions, collaboration and leadership on open source projects and initiatives within the open source ecosystem?"
@caniszczyk, @SuzanneA300, and others:
- I took Intel out of that list. But, if we add it back in then we should also include ARM
Here is a revised list.
AWS
Microsoft
Google
SAP
Salesforce
IBM / Red Hat
VMware
Facebook
Uber
Comcast
Verizon
Bloomberg
CapitalOne
While Red Hat is now owned by IBM, the open source practices of both companies are still very separate. I'd encourage not listing them as a single unit, particularly because community perceptions are likely to vary for them based on their separate histories.
Overall good. I like it. Lots of nuance things in the comments below.
On thing that stuck out for me was the relative emphasis on licensing and compliance over security. Historically licensing has been the hot topic but security is foremost on many people's minds. I'd like to have better understanding of how people are viewing security and what they are (or are not) doing about it.
Detailed comments:
Question 4
On first read it is ambiguous whether we're talking about how many products or how much open source is in each.
Question 5
- The "List of acceptable licenses" answer is out of place. It is a solution to the "use of open source" topic. Suggest removing it.
- The "employees are permitted" answers are similarly solutions rather than topics to be governed. Love the direction but suggest collapsing those into one something like "Allowing employees to contribute to non-work-related open source projects in their spare time". If we want more detail on strings being attached, perhaps a separate question.
Question 6
"Program" has proven to be a challenging word in past surveys. Many people are not familiar with it in the way we mean. Suggest either using a different term or backhandedly defining like
Does your company have a formal or informal management initiative or program around open source?
Question 7
Nit but suggest moving this later around the size and originating date questions (# 12 or so)
Question 10
Suggest adding an answer for "Security team" or some such. That would also need to go in 22
Question 11
Suggest using a different term for "Program Manager". At Microsoft we had 3 people whose title was "Program Manager". Suggest Program Lead or Program Director or something that implies the leader of the program. If changed, also update wording in answers for 23
Question 14
- Not sure what "Lower licensing fees" relates to. If this is the use of open source tools (e.g., Libre Office instead of Word) then I suggest removing it as not in the same vein as the other answers. Also update 24 and 26 and 30
Question 16
- I suggest an additional answer along the lines of "Getting engineering teams on board with compliance and security approaches"
- And another "Vulnerability monitoring and remediation" as a companion to License compliance overhead"
- Nit: The first answer starting with "Talent: " is out of place.
Question 26
Nit: Little odd having "Strategy:" as the only answer in that format
Question 27
Suggest defining out "program" again to avoid confusion similarly to my comment on 6. Perhaps:
Why doesn't your company have a formal or informal management initiative or program around open source?
Question 32
Very interesting.
- Merging IBM and Red Hat will lose considerable precision. I suspect that folks would answer quite differently if they were separate
- The list of companies is disturbingly US-centric
Question 33
Should clarify the roles here. perhaps
To what degree does a potential supplier's participation in, and contributions to, the open source community influence your organization’s buying decisions?
Question 34
- does "organization" mean "company" or "may part of the company"? The rest of the questions tend to use "company" when we mean "company".
- suggest "inside and outside" be "inside or outside"
Question 35
The meaning of this question is unclear. Is this asking about reallocation from in-person events to online/async material from open source folks?
Question 40
Reality check! need options for quarterly and annually. Most companies have yet to achieve devops release cadence.
Question 41
Do we want to include "and or DCO" in this? That is, why the focus on CLA? At the high level it is really "do y'all understand what it means to take contributions"? or is there something else we're after with this question?
Question 42
More of a curiosity: Do we care about the difference between 2 and 3 clause BSD? I'd think we cared more about GPL 2 vs GPL 3
Question 44
- Include OSS Review Toolkit
While Red Hat is now owned by IBM, the open source practices of both companies are still very separate. I'd encourage not listing them as a single unit
@mekkim I understand your point of view. If we ask about Red Hat separately, then we should also ask about GitHub separately. Both choices will almost assuredly get high ratings based on their long association with open source.
We are not asking "Microsoft / GitHub" (nor do I think we should) so the comparison is not the same. It really depends what we are trying to get out of this question. @LawrenceHecht, what is the goal of the question? If it is member brand recognition (for example) then we should be asking using the member's brand (e.g., IBM). If we want to get into all the subsidiaries, most of these companies have many well known subs.
@jeffmcaffer, the purpose of the question is to measure reputation and then see if it matters. I'm going to table discussion on this one question for now.
Question 39 - I would LOVE to see this paired with "How large is your company's engineering organization" and "How many employees does your company have"
We will look at the data based on # of employees
Also recommend asking how many employees contribute open source, rather than focusing only on developers. Season of Docs would be an argument for broadening the language here.
I understand your point. Non-developer contributions are important. That said, I think we should keep the question for two reasons: 1) to allow time series comparisons, and 2) this question already had a lot of people (17%) saying they don't know, and changing the question will increase that.
I'll go through @jeffmcaffer's comments tonight and make some changes to questions, a lot of the suggestions are good
@LawrenceHecht we should add a question regarding open source + security, essentially are people using automated tools to look at security issues on top of other concerns.
@caniszczyk I just made 2 pull requests:
#64 took care of some easy changes.
#65 has my recommendation for the final list of companies for the citizenship question.
Outstanding things I wanted to address are:
- moving #7 to after #12
- Update Q43, the question about " open source compliance methodologies and initiatives"
- adding a question about security. We have to be careful about asking about "automated tools" because most automation still requires manual work.
Chris, tomorrow I'll review your changes, with particular attention to how they many affect time series comparisons.
addressed a ton of @jeffmcaffer's concerned in 0a0adf9
@LawrenceHecht feel free to suggest any more changes but we are getting closer
Question 41
Do we want to include "and or DCO" in this? That is, why the focus on CLA? At the high level it is really "do y'all understand what it means to take contributions"? or is there something else we're after with this question?
@jeffmcaffer Last year, 16% required a CLA and 41% didn't know the answer to this question. My preference is to keep the question for time series purposes or get rid of it. If we did include DCO, I would prefer to break that out as separate from a CLA.
@caniszczyk & everyone else. I believe we addressed most people's comments. I am closing this issue. The next steps are copy editing, updating the survey's coding, and testing the survey before launching it.
Going to give the SC a bit more time, we're almost there
I'm all good after a final review this weekend.
@jeffmcaffer @DuaneOBrien @kpfleming @vmbrasseur any other thoughts here, I think we've improved the survey for this year after everyone's feedback.
LGTM
I think most if not all of what I commented on has been addressed. Thanks!