Data Collection and Analysis Challenges and Mitigation Strategies for Quantitative Human Factors Research Studies in Nuclear Power Plant Modernization
The United States (U.S.) Department of Energy (DOE) Light Water Reactor Sustainability (LWRS) program Plant Modernization Pathway is conducting targeted research and development (R&D) to address aging and reliability concerns with the legacy instrumentation and control and related information systems of the U.S. LWR fleet. In this effort, the application of human factors engineering (HFE) provides an important role in ensuring new digital plant technologies enable broad innovation and business improvement with continued operational safety. Evaluation is a key activity in HFE, which often occurs iteratively through the system design lifecycle. While qualitative methods are important in collecting information of how users perform tasks through observations, quantitative methods are equally important in assessing system design based on performance. In collecting and analyzing this quantitative data, there are notable challenges in the nuclear HFE domain that may threaten the validity and reliability of the inferences made in these studies. Notable challenges include small sample size and limited resources, large error variance and small effect size, an “adding test features to losing degrees of freedom” dilemma, non-normal distribution, and heterogeneity of variance. The results in control room usability studies are often statistically non-significant, which makes it hard to interpret. This work discusses these challenges across different scientific viewpoints and provides real-world examples of these challenges in practice. Collectively, the objective of this work is to position these challenges to the larger data science community as a means of identifying future opportunities to address these issues.