Prepare a reader survey
chriscool opened this issue · comments
In #625 we consider creating a reader survey to get feedback and learn how we could improve Git Rev News as edition 100 is fast approaching. Let's further discuss this idea here.
We could use @jnareb 's Survs.com account (from the time of Git User's Surveys - latest in 2016)
We should come up with a list of questions.
My personal opinion is that the story should remain - whether it's
short or long depends on what happened, but what IMHO counts
is its large variety and the chance to get an idea what's cooking even
if there is no time to follow the mailing list. In short: it keeps you
hungry ...
I concur. I've found it to be a crucial part of Rev News and have always enjiyed reading it. So, it would be great to have it as-is 🙂
@mjaix and @sivaraam thanks for your opinion!
I agree with moving the list of releases to the end.
I also agree with asking open questions in the survey. What do you think about the following ones:
- What are the Git Rev News sections you enjoy the most and why?
- What are the Git Rev News sections you don't enjoy much and why?
- What kind of information is the most useful to the community? Is it already available through Git Rev News? If not how should it appear?
- Does Git Rev News help recruit contributors to Git? How could it do better?
- Which Git Rev News section(s) would you like to appear at the top and which one(s) at the bottom?
- Which section(s) could evolve in a good way and how?
- Do you have ideas for new sections?
- Would you be interested in contributing to a section (existing or not) and why?
- How could Git Rev News help people who would like, or are starting, to contribute to Git?
- How could we make it easier to contribute to Git Rev News?
- How could we recruit new contributors or helpers to Git Rev News?
- Do you have suggestions for the Git Developer Pages website (https://git.github.io/)?
- How could Git Rev News get more readers?
I have a few questions to add related to interviews to your wonderful list of questions. Do include the ones which seem relevant.
- If there were a way for you suggest contributors to be interviewed, would it be helpful ?
- Is there anyone we've missed to interview ? / Is there anyone who you wish should be interviewed?
- We request for a re-interview of certain contributors who've been interviewed long back. Do you have any suggestions on contributors who you wish to be re-interviewed ?
- Do you have any suggestions on the set of interview questions ?
On another note, I believe we might need to categorize the questions a bit to ensure the survey doesn't seem too long 🤔
@sivaraam I like your great questions and I am Ok to add them as-is. Based on them I think we might want to add more questions related to each specific section we already have.
On the other hand, I am not sure categorizing is always a good idea. I would prefer to ask questions in a random order different for each reader, as I think answering a bunch of different questions first might give ideas for other answers to similar questions later.
A few other questions we might want to ask:
- What is missing in the "Releases" section? Which tools could/should we add to this section?
- How could we get more help with the scripts to automate the "Release" section and the publication of Git Rev News?
- How could we get more and better feedback from readers and the community?
@mjaix yeah, I am Ok with extending the "helpers" question. Thanks!
@jnareb any question related to links or any other question we could extend to cover links too?
@mjaix @sivaraam happy about the above questions so far?
I think we are close to having a fairly complete question list. What should we do now? Ask for feedback about the question list? Just create the survey and prepare to announce it in edition 99?
@jnareb any question related to links or any other question we could extend to cover links too?
Maybe a question about suggesting a blog to watch, or a good source of links (blog aggregator, newsletter, tech news site, not paywalled publishing service, etc.).
Also, what kind of links are preferred: tips and tricks, unusual use cases, new tools and workflows,...? Though I don't have good list of categories to ask about now - maybe make it open or semi open question?
@jnareb great! I have added the following questions related to the "Links" section to the question list :
- Do you know blogs we should watch, or a good source of links (blog aggregator, newsletter, tech news site, not paywalled publishing service, etc.) we should take advantage of?
- What kind of links do you prefer: tips and tricks, unusual use cases, new tools and workflows, research publications, videos, podcasts? Are there some kinds of links we aren't paying enough attention to?
- Do you know good ways to classify links that we could use?
thanks!
I have also made some changes to some questions and added a few new questions related to the "Discussions" section and the long articles there.
So here is the current list of questions then:
- What are the Git Rev News sections you enjoy the most and why ("Discussions", "Developer Spotlight" aka "Interview", "Releases", "Other News" aka "Links", "Credits")?
- What are the Git Rev News sections you don't enjoy much and why?
- What kind of information is the most useful and/or enjoyable to the community? Is it already available through Git Rev News? If not how should it appear?
- Does Git Rev News help recruit contributors to Git? How could it do better?
- If there was a way for you to suggest contributors to be interviewed, would it be helpful? If yes, what kind of way?
- Which Git Rev News section(s) would you like to appear at the top and which one(s) at the bottom?
- Which section(s) could evolve in a good way and how?
- About the discussions or sometimes news we summarize in the "Discussions" section towards the top of Git Rev News, what could we improve? Are the articles too long or too technical?
- Is there anyone we've missed to interview ? / Is there anyone who you wish should be interviewed?
- Do you have ideas for new sections or sub-sections?
- Do you know blogs we should watch, or a good source of links (blog aggregator, newsletter, tech news site, not paywalled publishing service, etc.) we should take advantage of?
- Would you be interested in contributing to a section (existing or not) and why?
- How could Git Rev News help people who would like, or are starting, to contribute to Git?
- We request for a re-interview of certain contributors who've been interviewed long back. Do you have any suggestions on contributors who you wish to be re-interviewed ?
- How could we make it easier to contribute to Git Rev News?
- What kind of links do you prefer: tips and tricks, unusual use cases, new tools and workflows, research publications, videos, podcasts? Are there some kinds of links we aren't paying enough attention to?
- How could we recruit new contributors, helpers or internal reviewers to Git Rev News?
- Do you have suggestions for the Git Developer Pages website (https://git.github.io/), its Git Rev News pages, its Hacking Git page, or its GSoC and Outreachy pages?
- How could Git Rev News get more readers?
- In the "Discussions" section towards the top, is the classification of articles between "General", "Review" and "Support" relevant? How could we improve on that?
- Do you have any suggestions on the set of interview questions ?
- What is missing in the "Releases" section? Which tools could/should we add to this section?
- How could we get more help with the scripts (in https://github.com/chriscool/getreleases) we use to automate the "Release" section and the publication of Git Rev News?
- Do you know good ways to classify links that we could use?
- Do you have suggestions about the email version of Git Rev News we send?
- How could we get more and better feedback from readers and the community?
I think I have added every question that was suggested, but maybe we could still improve this list a bit. Otherwise should we already prepare the survey?
Thanks @chriscool .
We should probably sort questions, and maybe divide them into sections, like "Spotlight" / "Interview", "Releases", etc.
@jnareb this was already suggested by @sivaraam above, but I have doubts (see #630 (comment)). I agree it might look better and perhaps easier to answer though. I also trust you on this as you have more survey experience than me.
So I am Ok with sorting questions and dividing them into sections if you prefer, but I am not sure it's easy to do in a nice way at least for me. For example I like that the last question is "How could we get more and better feedback from readers and the community?", but I think it would belong to a "General" section that would better be at the top of the survey.
Do you see a good way to sort and divide them into sections?
The current list of questions has around 30 questions - so I don't think splitting them into sections is necessary; having a list of questions all in one page, where you can easily go back to previous questions, is in my opinion a good thing.
However, having related questions close together reduces the number of "context switches". That is why I think we should "sort" questions.
What are the Git Rev News sections you enjoy the most and why ("Discussions", "Developer Spotlight" aka "Interview", "Releases", "Other News" aka "Links", "Credits")?
May be we should leave out the "Credits" part ? 🤔
Would you be interested in contributing to a section (existing or not) and why?
I suppose this is just an hypothetical question to know which sections others might be willing to contribute to. In that case, making it a bit explicit might help. May be phrasing it as follows helps?
If you were about to contribute to Git Rev News, which section would you be interested in contributing to (existing or not) and why?
Do you have suggestions about the email version of Git Rev News we send?
May be advertise more for its existence ? 😉 I really think we could include a line in each edition mentioning the possibility to get the newsletter via e-mail if they wish. It might help with better discovery of the same.
Just one suggestions for a question if we have room and fine it relevant:
How do you like the current design / layout of Git Rev News ? Have you noticed any issues with it ? Do you have any suggestions with improving it ?
IMHO the list is fairly complete, and I would also like to opt for a sorted structure staying on one page.
I too like the one page structure. I only have one concern about it, though. Given we have a long list of questions, would it not daunt the user who opens the survey and sees that it is a long one ? Would adding a prominent notice at the beginning mentioning that they are not compelled to answer every questions help reduce this to some extent ? 🤔
When we get around to the creation part, let me know if any help is needed. I would be glad to assist for the same. :-)
May be we should leave out the "Credits" part ?
Yeah, we can leave it out. On the other hand, keeping it might trigger a remark a reader would like to make about it.
If you were about to contribute to Git Rev News, which section would you be interested in contributing to (existing or not) and why?
Yeah, it seems to me a better wording than what I suggested.
I really think we could include a line in each edition mentioning the possibility to get the newsletter via e-mail if they wish. It might help with better discovery of the same.
Yeah, I would be Ok with adding such a line in each edition. I think we can still ask for suggestions about the email version.
How do you like the current design / layout of Git Rev News ? Have you noticed any issues with it ? Do you have any suggestions with improving it ?
Yeah, let's add this question too.
Given we have a long list of questions, would it not daunt the user who opens the survey and sees that it is a long one ? Would adding a prominent notice at the beginning mentioning that they are not compelled to answer every questions help reduce this to some extent ?
I think we can just say that all the questions are optional at the beginning. No need to make it more prominent than that.
Here is the current list of questions. I have split them into categories, but as previously mentioned I think we don't really need to categorize them in the survey, they could all be just on one page.
Introduction:
Thanks for helping us with improving Git Rev News by answering the following questions! We hope it will help us provide you a better newsletter starting with edition 100! All these questions are optional, and any constructive feedback is appreciated!
About the different sections in general:
- What are the Git Rev News sections you enjoy the most and why ("Discussions", "Developer Spotlight" aka "Interview", "Releases", "Other News" aka "Links", "Credits")?
- What are the Git Rev News sections you don't enjoy much and why?
- Which Git Rev News section(s) would you like to appear at the top and which one(s) at the bottom?
- Which section(s) could evolve in a good way and how?
- Do you have ideas for new sections or sub-sections?
About the "Discussion" section:
- About the discussions or sometimes news we summarize in the "Discussions" section towards the top of Git Rev News, what could we improve? Are the articles too long or too technical?
- In the "Discussions" section towards the top, is the classification of articles between "General", "Review" and "Support" relevant? How could we improve on that?
About the "Developer Spotlight" aka "Interview" section:
- If there was a way for you to suggest contributors to be interviewed, would it be helpful? If yes, what kind of way?
- Is there anyone we've missed to interview ? / Is there anyone who you wish should be interviewed?
- We request for a re-interview of certain contributors who've been interviewed long back. Do you have any suggestions on contributors who you wish to be re-interviewed ?
- Do you have any suggestions on the set of interview questions ?
About the "Other News" aka "Links" section:
- Do you know blogs we should watch, or a good source of links (blog aggregator, newsletter, tech news site, not paywalled publishing service, etc.) we should take advantage of?
- What kind of links do you prefer: tips and tricks, unusual use cases, new tools and workflows, research publications, videos, podcasts? Are there some kinds of links we aren't paying enough attention to?
- Do you know good ways to classify links that we could use?
About the "Releases" section:
- What is missing in the "Releases" section? Which tools could/should we add to this section?
About contributing to Git Rev News:
- If you were about to contribute to Git Rev News, which section would you be interested in contributing to (existing or not) and why?
- How could we make it easier to contribute to Git Rev News?
- How could we recruit new contributors, helpers or internal reviewers to Git Rev News?
- How could we get more help with the scripts (in https://github.com/chriscool/getreleases) we use to automate the "Release" section and the publication of Git Rev News?
General questions:
- What kind of information is the most useful and/or enjoyable to the community? Is it already available through Git Rev News? If not how should it appear?
- How could Git Rev News get more readers?
- Do you have suggestions for the Git Developer Pages website (https://git.github.io/), its Git Rev News pages, its Hacking Git page, or its GSoC and Outreachy pages?
- How could Git Rev News recruit and help people who would like, or are starting, to contribute to Git?
- How do you like the current design / layout of Git Rev News ? Have you noticed any issues with it ? Do you have any suggestions with improving it ?
- Do you have suggestions about the email version of Git Rev News we send?
- How could we get more and better feedback from readers and the community?
- Any other feedback?
I have added the introduction and the last question, and also squashed the 2 questions about recruiting and helping new contributors to Git into a single question.
Is everyone Ok with this?
@jnareb if there is no other suggestion soon, I think at this point I will leave it for you to create the survey.
@mjaix I agree with reducing the exclamation marks in the introduction. Thanks!
Started creating survey at https://survs.com/survey/jo07oxsaom
Started creating survey at https://survs.com/survey/jo07oxsaom
Filled in, there remains just the question of styling (and how to convert questions to survey forms).
In contrast to my previous statement, I am not so sure whether to leave only the last one or (maybe better) the first one.
FWIW, keeping the first one alone sounds like a good idea to me.
Started creating survey at https://survs.com/survey/jo07oxsaom
Filled in, there remains just the question of styling (and how to convert questions to survey forms).
About the styling, the font looks a bit too small on mobile. Is it possible to do something about it?
Screenshots for reference:
| Survey | GitHub |
|---|---|
Though I don't expect people to fill in survey answers on mobile. I wouldn't be sueprised of many take a peek at the questions on their mobile. If there's a quick way to tweak the fonts on mobile, it would be worth doing so. :-)
On a side note, it is good to keep the survey anonymous. OTOH, could we have an (optional) field asking for the respondent's email which we could reach out for further clarifications? That could be helpful for us. 🤔
About "We hope it will help us provide you a better newsletter starting with edition 100" in the introduction, we should likely say "101", instead of "100", if we announce the survey in edition 99, and "102" if we announce it in edition 100. That would leave one month for readers to complete the survey and one month for us to compile the results and start acting on them.
@mjaix and @chriscool Thanks for the suggestions. I have changed the survey text - please take a second look if I missed anything.
@sivaraam I don't think we can fix the styling on mobile when using third-party survey service like Survs.com.
I wonder if it would be worth it to split questions 8 and 9 into multiple choice part and free-form part:
- About the discussions or sometimes news we summarize in the "Discussions" section towards the top of Git Rev News, what could we improve?
- Are the articles too long or too technical?
- In the "Discussions" section towards the top, is the classification of articles between "General", "Review" and "Support" relevant?
- How could we improve on that?
Maybe not. This would also make the survey look longer, though...
Any preference to the theme used? I have switched from "Ocean" theme to "Graphite" theme - which I think looks more like the style used in the Git Rev News sent via e-mail as newsletter.
Should we add git logo?
@jnareb I think the survey is detailed and long enough now, so no need to split questions 8 and 9 further.
Adding the Git logo could make it look more friendly, so I think it would be great to do it if time permits.
I don't have a preference to the theme used.
Thanks again for the great work!
By the way, I was thinking about adding a question about usefulness of providing links to similar articles in past editions of Git Rev News, but then I realized that it would be a 'vanity' question (at least in a simple "Yes / No / Didn't notice" single-choice question), rather than providing something actionable, that can be used to improve Git Rev News.
Here is the survey: https://survs.com/survey/jo07oxsaom
Here is second channel, if one wanted separate link for e-mail version of Git Rev News Edition 99 : https://survs.com/survey/5nt0kyqhsa
Here is channel dedicated for social networks (Twitter, Mastodon, Bluesky, nostr, Facebook,...): https://survs.com/survey/8fk6jm88oj
I don't think there is any sense to provide embedded HTML version of the survey (HTML channel).
@jnareb do you know if we got a lot of feedback through the survey? Would it be worth sending an email to the mailing list to remind people that they only have a few weeks to complete the survey?
@jnareb do you know if we got a lot of feedback through the survey? Would it be worth sending an email to the mailing list to remind people that they only have a few weeks to complete the survey?
Currently Survs.com shows 6 completed responses.
I have not set any automatic cutoff date, so we can keep the survey open for more time.
@jnareb I am not sure keeping the survey open for more time will be enough to get other responses. I think people might just have forgotten about it now and won't even think about it if they are not reminded.
Yes, it might be worth it to remind people about Git Rev News Readers Survey.
6 responses is very small number. Which is strange, because every Git User's Survey (on Survs.com) had more than 3000 responses...
I just sent a reminder email: https://lore.kernel.org/git/CAP8UFD3ErGRbKSoM+_ExHSqiDe6=HgxCSttvmaMVLcpccZMjhg@mail.gmail.com/
I got a private reply to the reminder email saying "You probably shouldn't offer to re-interview meanwhile deceased people." Indeed should have have checked that. I think this concerns Shawn Pearce, but maybe others.
I'm trying to do my part of spreading news about the survey via social media (specifically Mastodon and Twitter; feel free to retweet/boost).
While doing that an idea occurred to me. The GitHub and GitLab social media profiles have a huge audience. Would it be make sense to ask a favour from them to spread the word about the survey (and consequently about Git RevNews too ;-) ?
I got a private reply to the reminder email saying "You probably shouldn't offer to re-interview meanwhile deceased people." Indeed should have have checked that. I think this concerns Shawn Pearce, but maybe others.
A rather unfortunate thing to consider (but a very valid one, nevertheless). I've been having an implicit criteria that the candidate being re-interviewed should be actively contributing to Git in recent times. Hopefully, that should be sufficient to ensure the concerned case doesn't happen.
I got a private reply to the reminder email saying "You probably shouldn't offer to re-interview meanwhile deceased people." Indeed should have have checked that. I think this concerns Shawn Pearce, but maybe others.
I'm sorry about this issue - when creating this question I have just extracted list of interviewed people wholesale, in chronological order, removing duplicates. I had hidden this answer in the survey from now on.
I just sent a reminder email: https://lore.kernel.org/git/CAP8UFD3ErGRbKSoM+_ExHSqiDe6=HgxCSttvmaMVLcpccZMjhg@mail.gmail.com/
This helped a bit - there are +2 more responses.
I just sent a reminder email: https://lore.kernel.org/git/CAP8UFD3ErGRbKSoM+_ExHSqiDe6=HgxCSttvmaMVLcpccZMjhg@mail.gmail.com/
This helped a bit - there are +2 more responses.
And now there are 10 responses in total.
Would posting a link to Git Rev News Readers Survey on the Git homepage be something that we might want to do?
@jnareb Sorry for not answering earlier. I think it might perhaps have been a good idea to do it when we launched the survey. On the other hand there are many people who are not readers who might have answered and that might have polluted the results. I think it's a bit too late now anyway.
This might not have been as much feedback as we wanted, but perhaps it's quality feedback, and it was worth it anyway to prepare and set this up.
I think I will send a last reminder when sending the draft for edition 100 hopefully this evening, and then we should close the survey on July 1st and just start analyzing the results.
For the sake of completeness, an initial analysis of the survey results could be found here [small related discussion].
Thanks all for your work on this! Closing it now.