This document describes a set of standards for what I consider to be reliable scientific research. I hold myself to these standards. While I do not necessarily expect others to agree, I do expect any research collaborators to adhere to these standards for projects where I am an author or co-author.
The standards are far from a complete description of valid scientific research, but they are my minimum set. They will continue to be developed and refined over time, so please feel free to discuss any concerns or disagreements with me. But please bring up any disagreements early, before a substantive time investment is made.
I'd be happy to provide reading or educational materials on any of these topics if you're unfamiliar with any of them.
The primary aim of an empirical research project is to gain a generalizable understanding of some objective truth. Multiple experiments to refine any explanations and rule out alternatives may be necessary.
The application and discovery of phenomena, while valuable, are secondary goals to explaining how something works, why some phenomenon occurs, and describing its extent and limits.
All human-subject research will comply with appropriate IRB (or local equivalent) guidelines. If the local institution does not have a body that is qualified or willing to regulate human-subjects research, then human-subjects research will not be conducted.
Everyone involved in collecting human-subject data will have passed a certification or course on responsible conduct in research (RCR) for human-subjects research, such as the CITI RCR courses.
Any experiment run prior to my joining a project will be considered part of a pilot study. It must be fine-tuned or simply replicated before being included in a manuscript.
To whatever extent is possible, decisions about data collection methods, analyses, and exclusion criteria will be made prior to looking at results. Undocumented HARKing (hypothesizing after results are known), "data peeking" (looking at data before determining the stopping point), and other Q.R.P.s (questionable research practices) will be avoided. For a list of example QRPs, see table 1.
If a preregistration is made for an experiment, and that experiment appears in a manuscript, it must be reported. No exceptions.
Any deviations from a preregistration will be noted in the manuscript.
To that end, all data collection and analyses will either report a preregistration or explicitly be described as "exploratory" or "descriptive".
All data collection methods will be clearly described, so that that they can be understood and replicated by others. All code, stimuli, questionnaires, etc. needed to replicate data collection will be shared. If any data collection materials cannot be shared (e.g., if the copyright belongs to other people), we will share as much as we can and state the reason for not sharing the rest in the paper.
All data collected for the manuscript will be shared in as raw a format as is reasonable and in an interoperable format such as CSV. Cleaned, processed, and aggregated data can be shared as a convenience to others, but the raw data must still be available.
This data will also include a clear description or "data dictionary" to facilitate reuse.
If ethical or practical concerns prevent sharing all of the data, we will share as much as we can and explain why the rest is not shared in the manuscript. However, concerns that prevent sharing should be strongly questioned if unreasonabe (an understandably subjective term). When possible, the unsahred data should be deposited in a protected-access repository.
All material and code needed for analysis will be shared. Some documentation for the process must be provided, but just stating which program to load it in may be sufficient.
All shared research material will be posted on a repository that is:
- Open access to allow free download and reuse (If privacy or other concerns prevent open access, a protected access repository will be used)
- Immutable and uniquely identifiable to ensure that a single consistent identifier will always lead to a single repository (versions may replace it if past versions remain discoverable and viewable)
- Has a clear plan for persistence to allow for reliable long-term access
Moreover, the artifacts will include a clear and open license for reuse with no restriction beyond an attribution requirement. CC0 and CC-BY (for text) and MIT (for code) are examples of licenses that meet this level of openness.
When possible, a preprint will be posted publicly prior to submission in order to get feedback from a broader audience than a few reviewers.
Intellectual property (IP) concerns may arise, especially for stimuli. While other people's intellectual property concerns may need to be protected, it is not acceptable to refuse to share our research project's artifacts that we collected or created due to our or our institution's intellectual property.
The publication will document the breakdown of contributions in accordance with CRediT (https://casrai.org/credit/). Using supplemental material may be necessary if the venue provides no official mechanism of denoting contribution.
Please feel free to reuse all of this document or just segments. Or feel free to use it as inspiration to make your own.
