slobentanzer / Reproducibility_HD2021-22

Reproducibility Seminar, Heidelberg University, Winter Semester 2021/22

Schedule

Note: We are back to Tue 5PM, online seminar

https://docs.google.com/document/d/198dnXWq8hGa68iw2TGABAy7DT3a6k9uk66Nh-h8oi4s/edit

Syllabus

Topics

• Background – Science as a societal endaevour
• Background – History of the scientific method
• Statistics – Common issues in the usage and interpretation of statistics
• Statistics – Frequentist models
• Related case study – Brian Wansink
• Statistics – Bayesian models
• Related case study – Sally Clark
• Statistics – Multiple testing
• Related case study – Biomarker development
• Practical - Code Peer Review
• Case study – Heart failure clinical trials (overestimation of effect sizes)
• Case study – Oxytocin (publication bias and null results)
• Case study – Jaques Benveniste (irrationality of the individual researcher and lack of retractions)
• Case study – John Ioannidis (cherry picking, academic misconduct)
• Background - Reproducibility in Bioinformatics
• Practical - Workflow Management
• Background – Reproducibility in AI

Recommended Reading

Journalistic Articles

1. https://en.wikipedia.org/wiki/Replication_crisis#In_medicine
2. https://www.nature.com/articles/d41586-021-01246-x (How COVID broke the evidence pipeline)
3. https://elifesciences.org/labs/dc5acbde/welcome-to-a-new-era-of-reproducible-publishing?utm_source=twitter&utm_medium=social
4. https://www.nature.com/articles/d41586-021-02242-x (Publication pressure: not industry, but governments?)
5. https://www.nature.com/articles/d41586-019-03350-5 (The need for better hypotheses)
6. https://www.chemistryworld.com/features/whats-wrong-with-research-culture/4014361.article (What is wrong with research culture?)

Blogs

1. https://lesslikely.com/statistics/s-values/ (P-Values Are Tough and S-Values Can Help)
2. https://rpsychologist.com/cohend/ (Interpretation of Cohen’s d)
3. http://daniellakens.blogspot.com/2017/12/understanding-common-misconceptions.html (Common misconceptions of p-values and how to avoid them)
4. https://www.bio.org/sites/default/files/legacy/bioorg/docs/Clinical%20Development%20Success%20Rates%202006-2015%20-%20BIO,%20Biomedtracker,%20Amplion%202016.pdf (Probability of success of clinical studies)

Scientific Articles

1. Ioannidis JP (2005) Why most published research findings are false. PLOS Medicine
2. Serra-Garcia M & Gneezy U (2021) Nonreplicable publications are cited more than replicable ones. Science Advances
3. Dougherty ER (2012) Biomarker development: Prudence, risk, and reproducibility. Bioessays
4. Scherer A (2017) Reproducibility in biomarker research and clinical development: a global challenge. Future Medicine
5. Gibson EW (2021) The role of p-values in judging the strength of evidence and realistic replication expectations. Statistics in Biopharmaceutical Research
6. Rafi Z & Greenland S (2020) Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise. BMC Medical Research Methodology
7. Heil BJ et al. (2021) Reproducibility standards for machine learning in the life sciences. Nature Methods
8. Gelman A & Carlin J (2014) Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science
9. Wratten L, Wilm A & Göke J (2021) Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers. Nature Methods
10. Hunter-Zinck et al. (2021) Ten simple rules on writing clean and reliable open-source scientific software. PLOS Computational Biology
11. Whalen S et al. (2021) Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics

Videos

1. https://youtu.be/_ArVh3Cj9rw (Vsauce, „The Future of Reasoning”)
2. https://youtu.be/HZGCoVF3YvM (3Blue1Brown, “Bayes’ Theorem“)
3. https://youtu.be/lG4VkPoG3ko (3Blue1Brown, “Why Bayes’ rule is nicer with odds“)

Case studies

Wansink case (slicing/p-hacking and HARKing):

1. https://web.archive.org/web/20170130155126/http://www.brianwansink.com/phd-advice/the-grad-student-who-never-said-no (i.e., almost everything that is wrong with how some labs are run)
2. https://peerj.com/preprints/2748/ (re-analysis finds more than 150 statistical inconsistencies in the four resulting papers alone)
3. https://statmodeling.stat.columbia.edu/2017/02/03/pizzagate-curious-incident-researcher-response-people-pointing-150-errors-four-papers-2/ (on the hubris of modern research)
4. https://www.motherjones.com/food/2018/09/cornell-food-researcher-brian-wansink-13-papers-retracted-how-were-they-published/ (conclusio)

Clark case (Bayesian stats, the prosecutor’s fallacy, and the ecological fallacy):

1. https://en.wikipedia.org/wiki/Sally_Clark (general overview, lots of more specific links, although some are dead)
2. http://plus.maths.org/issue21/features/clark/index.html (relation to Bayes)
3. https://understandinguncertainty.org/node/545 (very thorough analysis, fallacies)

Biomarker development (multiple testing)

1. Scherer (2017), see above; and references therein (particularly refs 1-5; please address the multiple testing aspect primarily)
2. Dougherty (2012), see above; and references therein

Heart failure clinical trials (on the overestimation of effect sizes)

1. Gibson (2021), see above; section 3 and references therein

Oxytocin makes people more trusting (on the difficulty of publishing null results and the ultimate reason for publication bias):

1. Positive result: https://www.nature.com/articles/nature03701, https://www.tandfonline.com/doi/full/10.1080/00207594.2012.677540
2. Negative result: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0137000
3. https://www.vox.com/2016/4/4/11348288/oxytocin-love-hormone

Benveniste case (homeopathic dilution of antibodies, on stubbornness, irrationality, and the sore lack of retractions):

1. https://www.bmj.com/content/329/7477/1290
2. Davenas et al. (1988), Nature, 'Human basophil degranulation triggered by very dilute antiserum against IgE' (original article)
3. Maddox et al. (1988), Nature, '"High-dilution" experiments a delusion' (rebuttal)

Ioannidis case (cherry picking, academic misconduct, political incentives?):

1. https://sciencebasedmedicine.org/what-the-heck-happened-to-john-ioannidis/
2. https://twitter.com/mendel_random/status/1308728096313536513

Cold fusion:

1. https://forbetterscience.com/2020/12/08/cold-fusion-by-eu-commission-a-fleischmann-pons-revival/

Further Reading

Books

1. Richard McElreath – Statistical Rethinking: A Bayesian Course with Examples in R and Stan
2. Edward R. Dougherty – The Evolution of Scientific Knowledge (available for free here: https://spie.org/Publications/Book/2263361?SSO=1)
3. Edward R. Dougherty, Michael L. Bittner – Epistemology of the Cell

Webpages

1. https://retractionwatch.com
2. https://pubpeer.com
3. https://forbetterscience.com
4. https://www.cos.io (Center for Open Science)

Null Results

1. The All Results Journals, http://arjournals.com/.
2. Journal of Articles in Support of the Null Hypothesis, http://www.jasnh.com/.
3. Journal of Pharmaceutical Negative Results, http://www.pnrjournal.com/.
4. Journal of Negative Results in BioMedicine, https://jnrbm.biomedcentral.com/.
5. Journal of Negative Results, http://www.jnr-eeb.org/index.php/jnr.

License

This work is licensed under Creative Commons CC-BY-NC 2.0.