Making sense of model generalizability: A tutorial on cross-validation in R and Shiny

Abstract

Model generalizability describes how well the findings from a sample are applicable to other samples in the population. In this tutorial, we first explain model generalizability through the statistical concept of model overfitting and its outcome (i.e., validity shrinkage in new samples), and provide a Shiny app to simulate and visually illustrate how it is influenced by three factors: model complexity, sample size, and effect size. We then discuss cross-validation as an approach for evaluating model generalizability, and provide brief guidelines for implementing this approach. Further, to help researchers understand how to apply cross-validation to their own research, we walk through an example, accompanied by step-by-step illustrations in R. This tutorial is expected to help readers develop the basic knowledge and skills to use cross-validation to assess model generalizability in their research and practice.

Publication
Advances in Methods and Practices in Psychological Science
Chen Tang
Chen Tang
PhD Student @ UIUC