Check for you affiliation and ORCID number and draft
rougier opened this issue · comments
Dear all,
We've just integrated affiliation and ORCID in the draft (PDF). Can you check your name/affiliation/ORCID is correct?
If not correct, you'll have to make correction on the parse-board.py script or on the 04-board.md file (PR welcome).
Also (especially for native English speakers), can you have a look at the draft and suggest corrections where needed?
Last, you can propose journal where to submit or vote (👍) for the one your prefer in the #6 issue.
Sorry for spam:
@rougier, @khinsen, @otizonaizit, @pdebuyl, @ctb, @ThomasA, @tpoisot, @karthik, @oliviaguest, @labarba, @MehdiKhamassi, @benoit-girard, @vitay, @gdetor, @dmcglinn, @yoavram, @heplesser, @apdavison, @neuronalX, @piero-le-fou, @mstimberg, @rossant, @eroesch, @damiendr, @delsuc, @soolijoo, @benureau, @rth, @almarklein, @pietromarchesi, @anyaevostinar, @ozancaglayan, @aaronshifman, @gviejo, @gdetor, @MehdiKhamassi, @benoit-girard, @opetchey, @Vahidrostami, @jsta, @Fjanks, @ChristophMetzner, @vitay, @falex33, @aaronshifman, @RafaelNH
Proposed journal: Science.
Added to #6
Affiliation and ORCID looks good.
I am about to submit a PR with some grammar fixes for the introduction.
I just finished a review of the paper draft. Look great!
I did have two thoughts:
-
In the GitHub Platform section I am worried that the terms
pull requestand contributionmergingwill be lost to people without git experience. Should we define them or rephrase? -
The ending is a bit of downer. Should we end with more optimism about ongoing educational efforts that may increase the frequency of replication? I seem to remember seeing activities by @opetchey in this area.
@jsta You're right. I would (personally) go for a definition (using box like for testimony ?) since it is als used on the website and such definition might help the curious reader. For the ending, yes, we could try to appear to be more optimistic but the reality is not so great concerning reproducibility.
I also had a look at the draft and agree that it looks great.
One comment: in the outlook it is stated that "ReScience is the only scientific journal that aims to verify the correctness of scientific software." and correctness is also mentioned in the introduction. Maybe it should be noted that a) the verification of correctness is not rigorous, i.e. the reviewers try their best to ensure correctness but no rigorous tests are applied, and b) that there are some efforts to automatically check for correctness of scientific models (e.g. the sciunit framework, https://github.com/scidash/sciunit, which is a framework that allows for validation of models against experimental observations and other models; the model validation framework within the Opensourcebrain project, https://github.com/OpenSourceBrain/osb-model-validation, which also allows for automated testing of models in the computational neuroscience field). Integration of such frameworks into the ReScience workflow could greatly enhance the verification of correctness and potentially speed up the review/replication process.
You're right, could propose a PR for attenuating the claim ?
Yes I can. I am pretty busy for the rest of this week, so could do it next week.
@oliviaguest The soft deadline is "before everyone's summer vacations", otherwise we'll lose the momentum we have.
Suggested timeline: substantial enhancements (new paragraphs etc.) by the end of June, final proofreading during the first week of July.
@khinsen OK, I have started reading and editing as I go (sent PR as done with 1st section) and will continue slowly before summer vacations of others (I have none planned — haha). 👍
Hi, affiliation is good for me, but please could you add my middle initial so that the name is Matt G Hall instead of Matt Hall.
Tx!
My affiliation and ORCID is fine.
I am in principle very happy with the draft, but for two issues:
- The manuscript points to PLOS' sharing policies. These are better than nothing, but they are not really enforced. Just this year I have read several new PLOS Comp Biol papers that did not even describe the details of the simulations behind the paper, not to speak of making software directly accessible. I think we might want to point to the difference between theory and practice.
- Drummond (2009) defined reproducibility and replicability exactly the opposite way as in the manuscript: mechanical "run it again" replication vs independent reproduction. Based on Drummond, Sharon Crook, Andrew Davison and I (2013) defined the following four categories:
- Internal replicability: The original author or someone in their group can re-create the results in a publication, essentially by rerunning the simulation software. For com- plete replicability within a group by someone other than the original author, espe- cially if simulations are performed months or years later, the author must use proper bookkeeping of simulation details using version control and electronic lab journals.
- External replicability: A reader is able to re-create the results of a publication using the same tools as the original author. As with internal replicability, all implicit knowledge about the simulation details must be entered into a permanent record and shared by the author. This approach also relies on code sharing, and readers should be aware that external replicability may be sensitive to the use of different hardware, operating systems, compilers, and libraries.
- Cross-replicability: The use of “cross” here refers to simulating the same model with different software. This may be achieved by re-implementing a model using a different simulation platform or programming language based on the original code, or by executing a model described in a simulator-independent format on different simulation platforms. Simulator-independent formats can be divided into declara- tive and procedural approaches. Assuming that all simulators are free of bugs, this would expose the dependence of simulation results on simulator details, but leads to questions about how to compare results.
- Reproducibility: Bob reads Alice’s paper, takes note of all model properties, and then implements the model himself using a simulator of his choice. He does not download Alice’s scripts. Bob’s implementation thus constitutes an independent implementation of the scienti c ideas in Alice’s paper based on a textual descrip- tion. The boundary line between cross-replicability and reproducibility is not always clear. In particular, a declarative description of a model is a structured, formalized version of what should appear in a publication, so an implementation by Charlie based on a declarative format might be considered to be just as independent as that by Bob based on reading the article.
I think this mainly shows how little defined terminology still is. But I think we should strive to stay with earlier uses of terms. It may very well be that Drummond and I are the odd ones out here, but I would be very interested to know how you arrived at your definitions.
@oliviaguest Yeah, I think "letter" is fine. Not sure the article as it stands fits the guidelines though.
@rougier I am going through the .tex btw. I'll try to submit pull requests as often as possible.
@heplesser We have had this discussion before, please #8.
Terminology is definitely not settled, and we won't settle it in this paper. The best we can do is define our choices, and be consistent with the use elsewhere in ReScience, in particular on the Web site. If you notice incoherent use of terms within the ReScience universe, please let us know!
I am reading the draft, and I believe the introduction is lacking a reference or two to the work of David Donoho, who has been an early and ardent advocate of reproducibility in science and in particular in mathematics and applied mathematics, as a member of the American Acad. of Science, and an active editor of PNAS, where he has been enforcing the policy of full access to code and data.
For instance, see "An invitation to reproducible computational research" David L. Donoho
Biostatistics (2010) 11 (3): 385-388. http://doi.org/10.1093/biostatistics/kxq028. (and ref therein)
@delsuc I agree. Can you make a PR to modify the draft accordingly?
I'll do that.
@khinsen @labarba My apologies for overlooking the R-words terminology discussion. BTW, what is the relation between the ReScience-article and the ReScience-article-2 repositories?
I think it would be useful to make explicit which terminological tradition this paper follows, e.g., by extending the last sentence of the first paragraph in the the section "Replication and reproduction" (the two references are just a suggestion):
"Here we briefly summarize the obstacles that arise from the use of computers and so ware in scientific research, and introduce the terminology we will use in the rest of this article, which follows the reproducible research movement (Claerbout and Karrenbach, 1992; Peng et al, 2006; ...)."
I think this will provide some clarity to readers, and document where the concepts in the paper originated.
I am still not entirely happy with the terminology; I will add my five cents to the R-words terminology discussion, as it seems to fit better there.
@heplesser ReScience-article was a first attempt to write an article about ReScience in a distributed/collaborative fashion. It led to a lot of useful discussion, but not to much progress towards a publishable article. ReScience-article-2 is a second attempt in which @rougier and myself have written a substantial draft before launching a call for contributions. We hope this will work out better, and so far I am optimistic.
Your proposition of explicitly referencing our terminological sources looks good - can you make a PR for this?
BTW, I am not happy with the current state of terminology either, and I suspect that noone really is. A look at the history of science shows that good terminology (and also good notations) takes a very long time to develop. We can console ourselves by claiming that we are participating in shaping history 😄 .
Now I am really confused. I just looked at the ReScience FAQ, which states
"Replication of a computational study means running the same computation on the same input data, and then checking if the results are the same, or at least “close enough” when it comes to numerical approximations. Replication can be considered as software testing at the level of a complete study.
Reproduction of a scientific study (computational or other) means repeating a published protocol, respecting its spirit and intentions but varying the technical details. For computational work, this would mean using different software, running a simulation from different initial conditions, etc. The idea is to change something that everyone believes shouldn’t matter, and see if the scientific conclusions are affected or not."
This seems rather the opposite of what the paper draft states:
"Reproducing the result of a computation means running the same software on the same input data and obtaining the same results ...
Replicating a published result means writing and then running new software based on the description of a computational model or method provided in the original publication, and obtaining results that are similar enough to be considered equivalent. ..."
We should be consistent between paper and FAQ.
@heplesser Good catch! It's the FAQ that needs updating, as the paragraph you quote is not in line with the rest of the site.
Oh f..., that's a (big) mistake. PR welcome.
@heplesser @rougier I just corrected the FAQ.
I am a bit puzzled by the fact that the "The ReScience Initiative" section states that a submission is accepted if reviewers "consider these results sufficiently close to the ones reported in the original paper being replicated.", leaving no explicit room for papers reporting replication failure, which is however presented as acceptable outcome by Fig. 1 "Success or failure to replicate is not a criterion for acceptance or rejection,".
@benoit-girard You're right because we accept failure to replicate, it's even in the FAQ.