Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Reproducibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Month-to-month and year-to-year reproducibility of high frequency QRS ECG signals

High frequency electrocardiography analyzing the entire QRS complex in the frequency range of 150 to 250 Hz may prove useful in the detection of coronary artery disease, yet the long-term stability of these waveforms has not been fully characterized. Therefore, we prospectively investigated the reproducibility of the root mean squared voltage, kurtosis, and the presence versus absence of reduced amplitude zones in signal averaged 12-lead high frequency QRS recordings acquired in the supine position one month apart in 16 subjects and one year apart in 27 subjects. Reproducibility of root mean squared voltage and kurtosis was excellent over these time intervals in the limb leads, and acceptable in the precordial leads using both the V-lead and CR-lead derivations. The relative error of root mean squared voltage was 12% month-to-month and 16% year-to-year in the serial recordings when averaged over all 12 leads. Reduced amplitude zones were also reproducible up to a rate of 87% and 81%, respectively, for the month-to-month and year-to-year recordings. We conclude that 12-lead high frequency QRS electrocardiograms are sufficiently reproducible for clinical use.

Coronary Disease/diagnosis↗

Perspectives on Data Reproducibility and Replicability in Paleoclimate and Climate Science

This paper summarizes the current state of reproducibility and replicability in the fields of climate and paleoclimate science, including brief histories of their development and applications in climate science, new and recent approaches towards improvement of reproducibility and replicability, and challenges. Recommendations for addressing those challenges include: development of searchable, auto-updated, interlinked, multi-archive public paleoclimate repositories for raw and processed digital datasets; cross-center standardized code base cases, improved data storage techniques, and a focus on replicability for climate simulation storage and access; and support of the development and community awareness of findable, accessible, interoperable and reusable (FAIR) principles by funding agencies and publishers. This paper is largely based on the May 2018 presentations of a panel of researchers to the Committee on Reproducibility and Replicability in Science, part of the National Academies of Science, Engineering, and Medicine. The commentary and recommendations made here are in alignment with those of its Consensus Study Report on Reproducibility and Replicability in Science (2019).

data repositories↗

Leveraging a Neural Network-Enhanced Reproducing Kernel Particle Method for Multiphysics Degradation Modeling of Energy Storage Materials

Energy storage materials exhibit strong electro-chemo-mechanical coupling and highly anisotropic material properties, contributing to the formation and propagation of micro-cracking during charge/discharge cycling and resulting in reduced performance and service life. A coupled electro-chemo-mechanical reproducing kernel particle method (RKPM) formulation has been developed to analyze this system. With microstructural images supplied by the National Renewable Energy Laboratory (NREL), pixel-based model construction by RKPM is used to represent the complex material microstructures that dictate the coupled physics of these systems. Traditional electro-chemo-mechanical models rely on mesh-based finite element methods, which can lead to difficulties in meshing such complex geometries and capturing crack propagation due to mesh dependency. Here, a neural network-enhanced reproducing kernel particle method (NN-RKPM) [1, 2] is introduced to effectively model damage and crack propagation in the material microstructures; the location, orientation, and solution transition near a localization are automatically captured by superimposed block-level NN optimizations. This NN enrichment approach allows for effective modeling of localizations via a fixed background discretization, relieving tedious efforts for adaptive refinement in traditional mesh-based methods. Applications to the heterogeneous microstructures of Li-ion battery cathodes will be presented to demonstrate the effectiveness of the proposed methods. NN-RKPM is additionally used to inform how crack opening and closure in turn affect the coupled chemical equations and material microstructure. Reference: [1] Baek, J., Chen, J. S., Susuki, K., "Neural Network enhanced Reproducing Kernel Particle Method for Modeling Localizations," International Journal for Numerical Methods in Engineering, Vol. 123, pp 4422-4454, https://doi.org/10.1002/nme.7040, 2022. [2] Baek, J., Chen, J. S., "A Neural Network-Based Enrichment of Reproducing Kernel Approximation for Modeling Brittle Fracture", Computer Methods in Applied Mechanics and Engineering Vol. 410, 116590, 2024.

degradation↗

Reproducibility of protein x-ray diffuse scattering and potential utility for modeling atomic displacement parameters

Protein structure and dynamics can be probed using x-ray crystallography. Whereas the Bragg peaks are only sensitive to the average unit-cell electron density, the signal between the Bragg peaks—diffuse scattering—is sensitive to spatial correlations in electron-density variations. Although diffuse scattering contains valuable information about protein dynamics, the diffuse signal is more difficult to isolate from the background compared to the Bragg signal, and the reproducibility of diffuse signal is not yet well understood. We present a systematic study of the reproducibility of diffuse scattering from isocyanide hydratase in three different protein forms. Both replicate diffuse datasets and datasets obtained from different mutants were similar in pairwise comparisons (Pearson correlation coefficient ≥0.8). The data were processed in a manner inspired by previously published methods using custom software with modular design, enabling us to perform an analysis of various data processing choices to determine how to obtain the highest quality data as assessed using unbiased measures of symmetry and reproducibility. The diffuse data were then used to characterize atomic mobility using a liquid-like motions (LLM) model. This characterization was able to discriminate between distinct anisotropic atomic displacement parameter (ADP) models arising from different anisotropic scaling choices that agreed comparably with the Bragg data. Our results emphasize the importance of data reproducibility as a model-free measure of diffuse data quality, illustrate the ability of LLM analysis of diffuse scattering to select among alternative ADP models, and offer insights into the design of successful diffuse scattering experiments.

59 BASIC BIOLOGICAL SCIENCES↗

Improved reproducibility of metal halide perovskite solar cells via automated gas quenching

Achieving reproducible perovskite solar cell fabrication is crucial for making it a scalable technology. We demonstrate an automated gas quenching system to improve perovskite solar cell reproducibility at the lab-scale. We use in situ photoluminescence to monitor the perovskite film formation as a function of the atmosphere in the glove box and find that antisolvent quenching is more sensitive to lingering precursor solvents than the gas quenching method. We observe a better reproducibility with gas quenching than with antisolvent quenching because it maintains a more consistent atmosphere in the glove box. The automated gas quenching process leads to high performing devices that are reproducible both batch to batch and researcher to researcher. The insights into gas quenching film formation as a function of solvent atmosphere and quench velocity will help inform future studies on large scale fabrication systems.

14 SOLAR ENERGY↗

HPC-driven computational reproducibility in numerical relativity codes: a use case study with IllinoisGRMHD

Abstract Reproducibility of results is a cornerstone of the scientific method. Scientific computing encounters two challenges when aiming for this goal. Firstly, reproducibility should not depend on details of the runtime environment, such as the compiler version or computing environment, so results are verifiable by third-parties. Secondly, different versions of software code executed in the same runtime environment should produceconsistent numerical results for physical quantities. In this manuscript, we test the feasibility of reproducing scientific results obtained using theIllinoisGRMHDcode that is part of an open-source community software for simulation in relativistic astrophysics, theEinstein Toolkit. We verify that numerical results of simulating a single isolated neutron star withIllinoisGRMHDcan be reproduced, and compare them to results reported by the code authors in 2015. We use two different supercomputers: Expanse at SDSC, and Stampede2 at TACC. By compiling the source code archived along with the paper on both Expanse and Stampede2, we find thatIllinoisGRMHDreproduces results published in its announcement paper up to errors comparable to round-off level changes in initial data parameters. We also verify that a current version ofIllinoisGRMHDreproduces these results once we account for bug fixes which have occurred since the original publication.

Astronomy & Astrophysics↗

Improving reproducibility in synchrotron tomography using implementation-adapted filters

For reconstructing large tomographic datasets fast, filtered backprojection-type or Fourier-based algorithms are still the method of choice, as they have been for decades. These robust and computationally efficient algorithms have been integrated in a broad range of software packages. The continuous mathematical formulas used for image reconstruction in such algorithms are unambiguous. However, variations in discretization and interpolation result in quantitative differences between reconstructed images, and corresponding segmentations, obtained from different software. This hinders reproducibility of experimental results, making it difficult to ensure that results and conclusions from experiments can be reproduced at different facilities or using different software. In this paper, a way to reduce such differences by optimizing the filter used in analytical algorithms is proposed. These filters can be computed using a wrapper routine around a black-box implementation of a reconstruction algorithm, and lead to quantitatively similar reconstructions. Use cases for this approach are demonstrated by computing implementation-adapted filters for several open-source implementations and applying them to simulated phantoms and real-world data acquired at the synchrotron. Our contribution to a reproducible reconstruction step forms a building block towards a fully reproducible synchrotron tomography data processing pipeline.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Reproducibility Practice in High-Performance Computing: Community Survey Results

The integrity of science and engineering research is grounded in assumptions of rigor and transparency on the part of those engaging in such research. HPC community effort to strengthen rigor and transparency take the form of reproducibility efforts. In a recent survey of the SC conference community, we collected information about the SC Reproducibility Initiative activities. Here, we present the survey results in this paper. Results show that the reproducibility initiative activities have contributed to higher levels of awareness on the part of SC conference technical program participants, and hint at contributing to greater scientific impact for the published papers of the SC conference series. Stringent point-of-manuscript-submission verification is problematic for reasons we point out, as are inherent difficulties of computational reproducibility in HPC. Future efforts should better decouple the community educational goals from goals that specifically strengthen a research work's potential for long-term impact through reuse 5-10 years down the road.

97 MATHEMATICS AND COMPUTING↗

MemXCT: Design, Optimization, Scaling, and Reproducibility of X-Ray Tomography Imaging

Here, this work extends our previous research entitled "MemXCT: Memory-centric X-ray CT Reconstruction with Massive Parallelization" that was originally published at SC19 conference (Hidayetoglu et al., 2019) with reproducibility of the computational imaging performance. X-ray computed tomography (XCT) is regularly used at synchrotron light sources to study the internal morphology of materials at high resolution. However, experimental constraints, such as radiation sensitivity, can result in noisy or undersampled measurements. Further, depending on the resolution, sample size and data acquisition rates, the resulting noisy dataset can be in the order of terabytes. Advanced iterative reconstruction techniques can produce high-quality images from noisy measurements, but their computational requirements have made their use an exception rather than the rule. We propose a novel memory-centric approach that avoids redundant computations at the expense of additional memory complexity. We develop a memory-centric iterative reconstruction system, MemXCT, that uses an optimized SpMV implementation with two-level pseudo-Hilbert ordering and multi-stage input buffering. We evaluate MemXCT on various supercomputer architectures involving KNL and GPU. MemXCT can reconstruct a large (11Kx11K) mouse brain tomogram in 10 seconds using 4096 KNL nodes (256K cores). The results presented in our original article at the SC19 were based on large-scale supercomputing resources. The MemXCT application was selected for the Student Cluster Competition (SCC) Reproducibility Challenge and evaluated on a variety of cloud computing resources by universities around the world in the SC20 conference. We summarize the results of the top-ranked SCC Reproducibility Challenge teams and identify the most pertinent measures for ensuring the reproducibility of our experiments in this article.

47 OTHER INSTRUMENTATION↗

A multimodal and integrated approach to interrogate human kidney biopsies with rigor and reproducibility: guidelines from the Kidney Precision Medicine Project

Comprehensive and spatially mapped molecular atlases of organs at a cellular level are a critical resource to gain insights into pathogenic mechanisms and personalized therapies for diseases. The Kidney Precision Medicine Project (KPMP) is an endeavor to generate three-dimensional (3-D) molecular atlases of healthy and diseased kidney biopsies by using multiple state-of-the-art omics and imaging technologies across several institutions. Obtaining rigorous and reproducible results from disparate methods and at different sites to interrogate biomolecules at a single-cell level or in 3-D space is a significant challenge that can be a futile exercise if not well controlled. Here we describe a “follow the tissue” pipeline for generating a reliable and authentic single-cell/region 3-D molecular atlas of human adult kidney. Our approach emphasizes quality assurance, quality control, validation, and harmonization across different omics and imaging technologies from sample procurement, processing, storage, shipping to data generation, analysis, and sharing. We established benchmarks for quality control, rigor, reproducibility, and feasibility across multiple technologies through a pilot experiment using common source tissue that was processed and analyzed at different institutions and different technologies. A peer review system was established to critically review quality control measures and the reproducibility of data generated by each technology before their being approved to interrogate clinical biopsy specimens. The process established economizes the use of valuable biopsy tissue for multiomics and imaging analysis with stringent quality control to ensure rigor and reproducibility of results and serves as a model for precision medicine projects across laboratories, institutions and consortia.

59 BASIC BIOLOGICAL SCIENCES↗

Open Reproducible Electron Microscopy Data Analysis

Electron microscopy (EM) is a cornerstone technique in the materials and biological sciences capable of imaging structures at nano- to atomic-scale resolution. Advances in technologies mean that one acquires datasets at increasing data rates and sizes. These advancements present enormous opportunities for researchers to understand complex systems. However, processing the resulting large-scale, complex data in a reproducible and shareable way is a real challenge for researchers. The building, managing, and maintaining complex workflows in a reproducible manner requires extensive knowledge in several areas outside the researchers’ core skill sets, such as software engineering, data science, and high-performance computing (HPC). Our work demonstrates an innovative approach to solving these problems, enabling reproducible EM data analysis through container encapsulated pipelines. Using modern container technologies, we encapsulate processing elements and connect them using shared memory. We expose user-friendly, advanced algorithms and tools to allow end users to utilize without expert programming skills. The platform enables reproducible, scalable, shareable pipelines for the analysis and visualization of EM data. Focusing on interoperability, we leverage the DOE and other agencies’ existing investments to provide a powerful software platform for EM data analysis.

Harris, Christopher↗

Using AI to Reproduce Neutrino Cross Section Analysis - Prototyping the Neutrino Discovery Platform

The Neutrino Discovery Platform (NDP) aims to accelerate DUNE-era science by making the neutrino program's existing datasets analyzable through fast, reproducible, and auditable workflows. We report a working version of two of its layers, data curation and agentic orchestration, built and tested end to end on MINERvA open data. The guiding lesson throughout is that a cross section is a measurement, and not just a plotted shape, only if it carries a defensible systematic-uncertainty budget, a trustworthy unfolding, and a reproducible record. Using a single medium-energy playlist pair from the MINERvA open-data release (about $2.05\times10^{17}$ protons on target of data), we first reproduced the shapes of two published charged-current inclusive $\nu_\mu$ measurements through a complete extraction ladder: selection, background subtraction, D'Agostini unfolding, efficiency correction, and flux normalization. These shape-level reproductions ran and tracked the published results, but they lacked the systematic-uncertainty machinery that defines a MINERvA cross section. To supply it, we vendored and built the MINERvA Analysis Toolkit and developed a many-universe systematic-uncertainty tool that produces a portable covariance artifact, a parallel event-loop runner, and a per-run auditability harness. Validated against a published covariance release, the toolchain reproduces the released statistical, flux, and muon-energy-scale terms and shows that they account for roughly 63\% of the total variance, with the remainder unreleased. Using this same infrastructure, we then performed a measurement of our own design, the hadronic recoil-energy distribution of low-energy ($E_\nu<2.5$~GeV) charged-current inclusive events, and found data/simulation shape agreement of $\chi^2/\mathrm{ndf}=1.26$. Together these results show that the platform supports original physics and not only reproductions.

Breaux, Auto [Tulane U. (main)]↗

Best practices in software development for robust and reproducible geoscientific models based on insights from the Global Carbon Budget's dynamic vegetation models

Computational models play an increasingly vital role in scientific research by enabling the numerical simulation of complex processes. Such models are also fundamental in geosciences. For instance, they offer critical insights into the impacts of global change on the Earth system today and in the future. Beyond their value as research tools, models are also software products and should therefore adhere to certain established software engineering standards. However, scientists are rarely trained as software developers, which can lead to potential deficiencies in software quality like unreadable, inefficient, or erroneous code. The complexity of models, coupled with their integration into broader workflows, also often makes it challenging to reproduce results, evaluate processes, and build upon them. In this paper, we review the state and current practices of the development processes of the state-of-the-art land surface models used by the Global Carbon Budget. We combine the experience of modelers from the respective research groups with the expertise of software engineers from tech companies to outline key principles and tools for improving software quality in research. We explore four main areas: (1) model testing and validation, (2) scientific, technical, and user documentation, (3) version control, continuous integration, and code review, and (4) the portability and reproducibility of workflows. Our review reveals that while modeling communities are incorporating many best practices, significant room for improvement remains in areas such as automated testing, automated documentation, and reproducibility. Therefore, we here identify and promote essential software engineering practices, including numerous examples of practices from within the community that can serve as guidelines for other models and could help streamline processes across the entire community. We conclude with an open-source example implementation of these principles, demonstrating portable and reproducible data flows, a continuous integration setup, and web-based visualizations. This example may serve as a practical resource for model developers, users, and all scientists engaged in scientific programming.

Gregor, Konstantin [Technical Univ. of Munich (Ger↗

Some Factors Affecting the Reproducibility of Penetration and the Cut-Off of Oil Sprays for Fuel-injection Engines

This investigation was undertaken at the Langley Memorial Aeronautical Laboratory in connection with a general research on fuel-injection for aircraft. The purpose of the investigation was to determine the factors controlling the reproducibility of spray penetration and secondary discharges after cut-off. The development of single sprays from automatic injection valves was recorded by means of special high-speed photographic apparatus capable of taking 25 consecutive pictures of the moving spray at a rate of 4,000 per second. The effect of two types of injection valves, injection-valve tube length, initial pressure in the injection-valve tube, speed of the injection control mechanism, and time of spray cut-off, on the reproducibility of spray penetration, and on secondary discharges were investigated. It was found that neither type of injection valve materially affected spray reproducibility. The initial pressure in the injection-valve tube controlled the reproducibility of spray penetrations. An increase in the initial pressure or in the length of the injection-valve tube slightly increased the spray penetration within the limits of this investigation. The speed of the injection-control mechanism did not affect the penetration. Analysis of the results indicates that secondary discharges were caused in this apparatus by pressure waves initiated by the rapid opening of the cut-off valve. The secondary discharges were eliminated in this investigation by increasing the length of the injection-valve tube. (author)

Beardsley, E G↗

Month-to-Month and Year-to-Year Reproducibility of High Frequency QRS ECG signals

High frequency (HF) electrocardiography analyzing the entire QRS complex in the frequency range of 150 to 250 Hz may prove useful in the detection of coronary artery disease, yet the long-term stability of these waveforms has not been fully characterized. We therefore prospectively investigated the reproducibility of the root mean squared (RMS) voltage, kurtosis, and the presence versus absence of reduced amplitude zones (RAzs) in signal averaged 12-lead HF QRS recordings acquired in the supine position one month apart in 16 subjects and one year apart in 27 subjects. Reproducibility of RMS voltage and kurtosis was excellent over these time intervals in the limb leads, and acceptable in the precordial leads using both the V-lead and CR-lead derivations. The relative error of RMS voltage was 12% month-to-month and 16% year-to-year in the serial recordings when averaged over all 12 leads. RAzs were also reproducible at a rate of up to 87% and 8 1 %, respectively, for the month-to-month and year-to-year recordings. We conclude that 12-lead HF QRS electrocardiograms are sufficiently reproducible for clinical use.

Batdorf, Niles↗

Using Docker Containers to Extend Reproducibility Architecture for the NASA Earth Exchange (NEX)

NASA Earth Exchange (NEX) is a data, supercomputing and knowledge collaboratory that houses NASA satellite, climate and ancillary data where a focused community can come together to address large-scale challenges in Earth sciences. As NEX has been growing into a petabyte-size platform for analysis, experiments and data production, it has been increasingly important to enable users to easily retrace their steps, identify what datasets were produced by which process chains, and give them ability to readily reproduce their results. This can be a tedious and difficult task even for a small project, but is almost impossible on large processing pipelines. We have developed an initial reproducibility and knowledge capture solution for the NEX, however, if users want to move the code to another system, whether it is their home institution cluster, laptop or the cloud, they have to find, build and install all the required dependencies that would run their code. This can be a very tedious and tricky process and is a big impediment to moving code to data and reproducibility outside the original system. The NEX team has tried to assist users who wanted to move their code into OpenNEX on Amazon cloud by creating custom virtual machines with all the software and dependencies installed, but this, while solving some of the issues, creates a new bottleneck that requires the NEX team to be involved with any new request, updates to virtual machines and general maintenance support. In this presentation, we will describe a solution that integrates NEX and Docker to bridge the gap in code-to-data migration. The core of the solution is saemi-automatic conversion of science codes, tools and services that are already tracked and described in the NEX provenance system, to Docker - an open-source Linux container software. Docker is available on most computer platforms, easy to install and capable of seamlessly creating and/or executing any application packaged in the appropriate format. We believe this is an important step towards seamless process deployment in heterogeneous environments that will enhance community access to NASA data and tools in a scalable way, promote software reuse, and improve reproducibility of scientific results.

earth exchange↗

Explaining and predicting human behavior and social dynamics in simulated virtual worlds: reproducibility, generalizability, and robustness of causal discovery methods

Ground Truth program was designed to evaluate social science modeling approaches using simulation test beds with ground truth intentionally and systematically embedded to understand and model complex Human Domain systems and their dynamics Lazer et al. (Science 369:1060–1062, 2020). Our multidisciplinary team of data scientists, statisticians, experts in Artificial Intelligence (AI) and visual analytics had a unique role on the program to investigate accuracy, reproducibility, generalizability, and robustness of the state-of-the-art (SOTA) causal structure learning approaches applied to fully observed and sampled simulated data across virtual worlds. In addition, we analyzed the feasibility of using machine learning models to predict future social behavior with and without causal knowledge explicitly embedded. In this paper, we first present our causal modeling approach to discover the causal structure of four virtual worlds produced by the simulation teams—Urban Life, Financial Governance, Disaster and Geopolitical Conflict. Our approach adapts the state-of-the-art causal discovery (including ensemble models), machine learning, data analytics, and visualization techniques to allow a human-machine team to reverse-engineer the true causal relations from sampled and fully observed data. We next present our reproducibility analysis of two research methods team’s performance using a range of causal discovery models applied to both sampled and fully observed data, and analyze their effectiveness and limitations. We further investigate the generalizability and robustness to sampling of the SOTA causal discovery approaches on additional simulated datasets with known ground truth. Our results reveal the limitations of existing causal modeling approaches when applied to large-scale, noisy, high-dimensional data with unobserved variables and unknown relationships between them. We show that the SOTA causal models explored in our experiments are not designed to take advantage from vasts amounts of data and have difficulty recovering ground truth when latent confounders are present; they do not generalize well across simulation scenarios and are not robust to sampling; they are vulnerable to data and modeling assumptions, and therefore, the results are hard to reproduce. Finally, when we outline lessons learned and provide recommendations to improve models for causal discovery and prediction of human social behavior from observational data, we highlight the importance of learning data to knowledge representations or transformations to improve causal discovery and describe the benefit of causal feature selection for predictive and prescriptive modeling.

97 MATHEMATICS AND COMPUTING↗

Rigor and Reproducibility in Electrocatalysis: Best Practices for Operando Studies

Operando measurements have rapidly expanded the scope of electrocatalysis by enabling direct observation of catalytic interfaces under working conditions and by linking structural, compositional, and spectroscopic observables to activity and selectivity. However, the growth of operando methods has outpaced the adoption of broadly shared experimental standards, creating persistent challenges in reproducibility, interpretation, and comparison across laboratories and platforms. This perspective synthesizes discussions from the 2025 National Science Foundation Workshop on Rigor and Reproducibility in Electrocatalysis and outlines a practical framework for the rigorous use of operando measurements in electrocatalysis. We highlight three recurring needs: careful implementation of complex methods to avoid overinterpretation; recognition that (subtle) differences in reactor architecture, hydrodynamics, and electrical boundary conditions can alter apparent kinetics and selectivity; and transparent reporting standards that enable meaningful cross-comparison without constraining measurement-specific cell innovation. Focusing on widely used techniques (including X-ray and vibrational spectroscopies, mass spectrometry, and electron microscopy), we discuss technique-specific pitfalls, cross-validation strategies, and recurring platform-agnostic considerations such as mass transport, current distribution, temporal-resolution mismatches, and catalyst evolution. This Perspective aims to strengthen the mechanistic inference and improve the reproducibility, comparability, and predictive value of operando electrocatalysis research.

X-ray absorption spectroscopy↗