"Developing and validating a comprehensive implementation framework for reporting reproducible infectious disease computational modeling studies" - PUblic Health/Epidemiology

Committee:
Marquis Hawkins, Department of Epidemiology  (advisor and dissertation director)

Harry Hochheiser, Department of Biomedical Informatics, School of Medicine 

Bruce Childers, Department of Computer Science, School of Computing and Information 

Mark Roberts, Department of Health Policy and Management 

Abstract: 
In the wake of the coronavirus disease 2019 (COVID-19) pandemic, policymakers have relied heavily on computational models to inform decisions concerning public health interventions. Unfortunately, reproducibility of computational modeling studies is limited due to methodological complexity and lack of transparent reporting practices. We filled this critical gap in the literature by developing an implementation framework for representing infectious disease computational models in a reproducible format, grounded in previous research on reproducibility from a broad range of scientific disciplines. The implementation framework provides a foundation that can be further developed into tools, such as checklists or machine-interpretable metadata, for sharing computational models in a reproducible manner.  

We formatted the implementation framework into the Infectious Disease Modeling Reproducibility Checklist (IDMRC) and validated the checklist through an iterative process by evaluating random samples of infectious disease modeling studies. In addition to our framework and the IDMRC, we evaluated the adoptability of workflow tools, for representing, evaluating, and reproducing models which may lead to useful insights for improving the coordination of modeling resources. We tested the feasibility of reproducing a COVID-19 modeling study using the Open Curation for Computer Architecture Modeling (Occam), an open-source workflow platform that encapsulates and preserves the complete experimental workflow of a modeling study. 

For years, attempts have been made to develop a comprehensive tool that can be adopted by researchers, journal editors, and scientific organizations with minimal success in preventing irreproducible models from being published. Our implementation framework and the IDMRC are the first reproducibility tools that can be used by researchers to assess infectious disease computational modeling studies starting from the description of the model and ending at the obtainment of the results. By easily comparing models and their output, researchers will be able to efficiently identify the best models to inform life-saving interventions.