Cynthia Thomes is a reference and instruction librarian at University of Maryland University College (UMUC).
Cynthia won a travel grant from JoVE to attend the Charleston Conference and will be speaking about the importance of reproducibility in systematic reviews as part of the panel ‘Librarians Leading the Way to Improved Research Reproducibility’ on Nov. 8th in Charleston.
Ed. Note: This interview has been lightly edited for clarity.
JoVE: How do you define the ‘Reproducibility Problem’?
Cynthia Thomes: I think it’s important to make it very clear that research reproducibility is important in the social sciences as well as in the natural sciences. The process of attempting to reproduce another person’s research can help to identify any previously undiscovered problems with the methodology used for the original research study and/or with that study’s conclusions.
J: Can librarians positively impact the research process?
CT: Librarians are uniquely positioned to help support research reproducibility since many of us have contacts in many of the different communities involved in the research process. One way to have an impact is by being more involved at an earlier stage in the research process. When students are conducting background reading to better understand their research topic, for instance, they may not know which databases to use to research their particular topic, or they may not be able to construct effective search statements in order to find the articles and other documents that best discuss their topic. Once they’ve identified relevant documents, they may not know how to conduct cited reference searching in order to find more recent documents on their topic -- documents that may, in fact, have attempted to confirm the findings of the original document of interest.
J: How did you personally get involved?
CT: I am my library’s sole liaison to the university’s doctoral programs department, and in this role, I work very closely with students at all stages of their dissertation research. All doctoral students’ dissertations currently take the form of systematic reviews, which means that reproducibility plays a very important role in the students’ work.
Librarian Reference Team at the University of Maryland University College (UMUC)
J: Why is reproducibility important to systematic reviews?
CT: Although research reproducibility is probably much more closely associated with the collection of primary data, it’s important that students’ systematic review research be reproducible since their dissertations’ conclusions and recommendations often end up being used as the basis for decisions made at the students’ workplaces. Doctoral faculty and I make it clear to students that since their research results may be used to inform real-world decisions, it’s crucial that others who read their dissertations be able to find the same set of documents that the students found and that they come to the same conclusions.
J: What steps of systematic reviews directly affect its reproducibility?
CT: Systematic reviews require students to meticulously document each step of their research process: which databases they used, what search strings they entered into those databases, how many results they retrieved, which database limiters they applied to their search results, what other inclusion/exclusion criteria they applied to their results, etc. This documentation is intended in part to ensure that students’ work can be replicated by others. Indeed, as Hemingway and Brereton (2009) put it, “Systematic reviews attempt to bring the same level of rigor to reviewing research evidence as should be used in producing that research evidence in the first place”.
J: How do you connect with students and faculty on this topic?
CT: I have worked to make students and faculty aware of useful books, articles, websites, and other resources that explain the systematic review process. I’ve sent e-mail messages, posted in online classrooms, given presentations at students’ required residencies, to name a few. I’ve also educated students and faculty about resources such as RefWorks and ReadCube that they can use to keep track of the documents that they find during the course of their research.
J: Do you think new tools are helpful to research reproducibility?
CT: With any luck, new technologies will make it even easier for researchers to document the key steps that they’ve taken and crucial decisions that they’ve made during the course of their research, which should make their research that much easier to reproduce and confirm.
With regard to systematic reviews, for instance, the work of Rob Capra and Jaime Arguello to develop systems that display relevant “search trails” to researchers seems promising!
Want to receive some useful reads and resources about reproducibility?