SEARCH · Engineering Papers

Results for “Reproducibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Towards modelling AR Sco: calibration – reproducing high-energy pulsar emission and testing convergence to Aristotelian electrodynamics

In recent years, kinetic simulations have been crucial to further our understanding of pulsar electrodynamics. Yet, due to the large-scale separation between the gyro-period and the stellar rotation period, resolving the particle gyration has been computationally unfeasible for realistic pulsar parameters. The main aim of this work is comparing our gyro-phase-resolved model with a gyro-centric pulsar model, where our model solves the general equations of motion with included radiation reaction using a higher order numerical solver with adaptive time-steps. Specifically, we aim to (i) reproduce a pulsar’s high-energy emission maps, namely one with 10 per cent of the surface B-field strength of Vela, and the spectra produced by an independent gyro-centric pulsar emission model; and (ii) test convergence of these results to the radiation-reaction limit of Aristotelian electrodynamics. (iii) Additionally, we identify the effect that a large $E_{\parallel }$-field has on the trajectories and radiation calculations. We find that we can reproduce the curvature radiation emission maps and spectra well, using 10 per cent field strengths of the Vela pulsar and injecting our particles at a higher altitude in the magnetosphere. Using sufficiently large $E_{\parallel }$-fields, our numeric results converge to the analytic radiation-reaction limit trajectories. Additionally, we illustrate the importance of accounting for the $\mathbf {E}\times \mathbf {B}$-drift in the particle trajectories and radiation calculations, validating the Harding and collaborators’ model approach. Lastly, we found that our model deals very well with the high-radiation-reaction and high-field regimes present in pulsars.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2023

Building the I (Interoperability) of FAIR for performance reproducibility of large-scale composable workflows in RECUP

Abstract-Scientific computing communities increasingly run their experiments using complex data- and compute-intensive workflows that utilize distributed and heterogeneous architectures targeting numerical simulations and machine learning, often executed on the Department of Energy Leadership Computing Facilities (LCFs). We argue that a principled, systematic approach to implementing FAIR principles at scale, including fine-grained metadata extraction and organization, can help with the numerous challenges to performance reproducibility posed by such workflows. We extract workflow patterns, propose a set of tools to manage the entire life cycle of performance metadata, and aggregate them in an HPC-ready framework for reproducibility (RECUP). We describe the challenges in making these tools interoperable, preliminary work, and lessons learned from this experiment.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Managing Software Provenance to Enhance Reproducibility in Computational Research

Scientific processes rely on software as an important tool for data acquisition, analysis, and discovery. Over the years, sustainable software development practices have made progress in being considered as an integral component of research. However, management of computation-based scientific studies is often left to individual researchers who design their computational experiments based on personal preferences and the nature of the study. Here, we believe that the quality, efficiency, and reproducibility of computation-based scientific research can be improved by explicitly creating an execution environment that allows researchers to provide a clear record of traceability. This is particularly relevant to complex computational studies in high-performance computing (HPC) environments. In this article, we review the documentation required to maintain a comprehensive record of HPC computational experiments for reproducibility. We also provide an overview of tools and practices that we have developed to perform such studies around Flash-X, a multiphysics scientific software.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

QAOAKit: A Toolkit for Reproducible Study, Application, and Verification of the QAOA

Understanding the best known parameters, performance, and systematic behavior of the Quantum Approximate Optimization Algorithm (QAOA) remain open research questions, even as the algorithm gains popularity. We introduce QAOAKit, a Python toolkit for the QAOA built for exploratory research. QAOAKit is a unified repository of preoptimized QAOA parameters and circuit generators for common quantum simulation frameworks. We combine, standardize, and cross-validate previously known parameters for the MaxCut problem, and incorporate this into QAOAKit. We also build conversion tools to use these parameters as inputs in several quantum simulation frameworks that can be used to reproduce, compare, and extend known results from various sources in the literature. We describe QAOAKit and provide examples of how it can be used to reproduce research results and tackle open problems in quantum optimization.

Shaydulin, Ruslan↗

DOE OSTI2020

A Python Instrument Control and Data Acquisition Suite for Reproducible Research

Tools that standardize and automate experimental data collection are needed for greater confidence in research results. The National Synchrotron Light Source-II (NSLS-II) has generated an open-source Python data acquisition, management, and analysis software suite that automates X-ray experiments and collects an experimental record that facilitates complete reproducibility. Here, we show that the NSLS-II tools are not only useful for X-ray science at large-scale facilities by presenting an add-on package that adapts these tools for use in a small laboratory with common physics and electrical engineering instruments. The composite software suite eases and automates the execution of experiments, records extensive metadata, stores data in portable containers, and speeds up the analysis through tools for comprehensive searches. In total, this software suite increases the reproducibility of laboratory experiments. We demonstrate the software via the evaluation of two lock-in amplifiers-the miniature ADA2200 and the ubiquitous Stanford Research Systems (SRS) SR810. The frequency resolution, signal-to-noise ratio, and dynamic reserve of the lock-in amplifiers are measured and presented. The usage of the software suite is described throughout these measurements so that the reader can implement the tools in their lab.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Science Capsule: Towards Sharing and Reproducibility of Scientific Workflows

Workflows are increasingly processing large volumes of data from scientific instruments, experiments and sensors. These workflows often consist of complex data processing and analysis steps that might include a diverse ecosystem of tools and also often involve human-in-the-loop steps. Sharing and reproducing these workflows with collaborators and the larger community is critical but hard to do without the entire context of the workflow including user notes and execution environment. In this paper, we describe Science Capsule, which is a framework to capture, share, and reproduce scientific workflows. Science Capsule captures, manages and represents both computational and human elements of a workflow. It automatically captures and processes events associated with the execution and data life cycle of workflows, and lets users add other types and forms of scientific artifacts. Science Capsule also allows users to create `workflow snapshots' that keep track of the different versions of a workflow and their lineage, allowing scientists to incrementally share and extend workflows between users. Our results show that Science Capsule is capable of processing and organizing events in near real-time for high-throughput experimental and data analysis workflows without incurring any significant performance overheads.

Ghoshal, Devarshi↗

DOE OSTI2023

Materials characterization: Can artificial intelligence be used to address reproducibility challenges?

Material characterization techniques are widely used to characterize the physical and chemical properties of materials at the nanoscale and, thus, play central roles in material scientific discoveries. However, the large and complex datasets generated by these techniques often require significant human effort to interpret and extract meaningful physicochemical insights. Artificial intelligence (AI) techniques such as machine learning (ML) have the potential to improve the efficiency and accuracy of surface analysis by automating data analysis and interpretation. In this perspective paper, we review the current role of AI in surface analysis and discuss its future potential to accelerate discoveries in surface science, materials science, and interface science. We highlight several applications where AI has already been used to analyze surface analysis data, including the identification of crystal structures from XRD data, analysis of XPS spectra for surface composition, and the interpretation of TEM and SEM images for particle morphology and size. We also discuss the challenges and opportunities associated with the integration of AI into surface analysis workflows. These include the need for large and diverse datasets for training ML models, the importance of feature selection and representation, and the potential for ML to enable new insights and discoveries by identifying patterns and relationships in complex datasets. Most importantly, AI analyzed data must not just find the best mathematical description of the data, but it must find the most physical and chemically meaningful results. In addition, the need for reproducibility in scientific research has become increasingly important in recent years. The advancement of AI, including both conventional and the increasing popular deep learning, is showing promise in addressing those challenges by enabling the execution and verification of scientific progress. By training models on large experimental datasets and providing automated analysis and data interpretation, AI can help to ensure that scientific results are reproducible and reliable. Although integration of knowledge and AI models must be considered for the transparency and interpretability of models, the incorporation of AI into the data collection and processing workflow will significantly enhance the efficiency and accuracy of various surface analysis techniques and deepen our understanding at an accelerated pace.

Materials Science↗

DOE OSTI2022

Reproduced Computational Results Report for “Ginkgo: A Modern Linear Operator Algebra Framework for High Performance Computing”

The article titled “Ginkgo: A Modern Linear Operator Algebra Framework for High Performance Computing” by Anzt et al. presents a modern, linear operator centric, C++ library for sparse linear algebra. Experimental results in the article demonstrate that Ginkgo is a flexible and user-friendly framework capable of achieving high-performance on state-of-the-art GPU architectures. In this report, the Ginkgo library is installed and a subset of the experimental results are reproduced. Specifically, the experiment that shows the achieved memory bandwidth of the Ginkgo Krylov linear solvers on NVIDIA A100 and AMD MI100 GPUs is redone and the results are compared to what presented in the published article. Upon completion of the comparison, the published results are deemed reproducible.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Experimental Benchmarking for High-Reproducibility, Cross-Institutional Evaluation of Iron Redox Electrochemistry

We present a practical case study standardizing experimental protocols between collaborators with the goal of understanding ferrous iron (Fe 2+ ) chemistry and improving the iron deposition reaction for energy-efficient, electrochemical iron production. The study of iron reactions can be difficult, as aqueous iron electrolytes exhibit complex behaviors that can lead to differing interpretation of ostensibly similar experiments. The question we want to answer: are we studying the same chemistry? Our protocols address inherent challenges such as the tendency for Fe 2+ to spontaneously oxidize to ferric iron (Fe 3+ ) and the production of hydrogen at the potentials of interest. Our standardized protocol, executed by four collaborators in different labs and institutions, yields high-reproducibility results, and identified glassy carbon electrode surface quality and Fe 3+ impurities in the salt as key factors with outsized effects on cyclic voltammetry measurements. The process of developing the protocols helped to troubleshoot underlying issues that created poorer reversibility and reproducibility. This study highlights the fact that even nominally straightforward electrochemical systems can yield vastly different outcomes due to small differences in experimental preparation and serves as a useful example for creating transparent and achievable standards for the generation of reliable datasets that can be widely used and shared.

Ketter, Benjamin [Argonne National Laboratory (ANL↗

DOE OSTI2025

Videos, photos, and AI-derived grain size data associated with “High-throughput AI Video Surveys Enable Reproducible Multiscale Sediment Size Mapping, with Implications for Hydrobiogeochemical Parameterization”

NOTE: The manuscript associated with this data package is currently in review. The data may be revised based on reviewer feedback. Upon manuscript acceptance, this data package will be updated with the final dataset and additional metadata. This data package is associated with the manuscript “High-throughput AI Video Surveys Enable Reproducible Multiscale Sediment Size Mapping, with Implications for Hydrobiogeochemical Parameterization” under review. This data package includes five data types: 1) raw photos and videos from drone survey and walking smartphone surveys; 2) images derived from raw videos; 3) manual labeling of reference scales; 4) metadata for all images and photo resolution derived from artificial intelligence (AI) models or manual labels, 5) grain size data obtained from AI models for all photos, 6) metadata and grain size data after quality control, 7) summaries of sample efficiency for all data, and 8) computational fluid dynamics (CFD) data used to support hydro-biogeochemical (HBGC) parameter estimation. Such data is used to 1) demonstrate significant improvements in accuracy, efficiency, and quality control for grain size data collection with the help of AI models, 2) study the spatial heterogeneity of grain size and observation reproducibility based on tens of thousands of data points generated by the AI models, and 3) evaluate the impacts of grain size heterogeneity on key HBGC parameters across sediment-to-reach and hourly-to-yearly scales. In particular, the data package contains 116 folders and 179696 files. The files include 41 videos in .mov format, 64047 photos in .jpg format, 13541 video-derived photos in .png format, 12747 segmentation mask data in .tif format, 12747 segmentation data in .json format, 24771 .csv files that with metadata and grain size for each individual photo as well as water depth and velocity data from CFD and observation, 51791 .txt files of raw AI predicted labels, and 11 flight record data in .srt format. The summary for all metadata and grain size statistics information is included in “Scales_V3_NG.csv” and “Statistics_V3_NG.csv”. The summary for data that pass data quality control (QC) level 0-2 is included in “QCStatistics_V3_NG.csv”. The QC level 0 represents photos whose photo resolution is positive, excluding photos that miss reference scale. The QC level 1 means reference scale circularity uncertainty is less than 5% for smartphone images while representing photo resolution is larger than 0.44 mm/pixel for drone images. The QC level 2 means excluding photos whose grain number is less than 100, a minimum number of grains recommended by classic literature. The summary for each video’s name, length, frame rates, survey area, grain number, survey efficiency, etc. can be found in “QCSummary_V3_NG.csv”. The summary for site name, GPS coordinates, and number of images at each site can be found in “SitesSummary_V3_*.csv” files. Overall computational efficiency summary is reported in Table 4 of accompanying manuscript. Additionally, the nitrate concentration data used in this work was downloaded from an existing dataset published on ESS-DIVE (Boat-Dragged Sensor Hanford Reach.csv; Conner A. et al., 2020). We thank the United States Forest Service, Washington Department of Fish and Wildlife, Washington Department of Natural Resources, Cowiche Canyon Conservatory, Port of Benton, and the Confederated Tribes and Bands of the Yakama Nation for access to field locations where the data were collected. We also thank the Yakama Nation Tribal Council and Yakama Nation Fisheries for working with us to facilitate data collection and optimization of data usage according to their values and worldview.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

A reproducible study design for the MIMIC-IV in-hospital mortality task

Open, tabular electronic health record (EHR) datasets such as MIMIC-III and MIMIC-IV have become critical resources for developing machine learning (ML) models addressing clinical prediction tasks, including hospital readmission, length of stay, and in-hospital mortality (IHM). While MIMIC-III has benefited from well-established preprocessing pipelines and standardized feature sets, MIMIC-IV remains comparatively challenging to work with because there are no standardized benchmarks to support reproducibility and comparability across studies. To address this limitation, we present a rigorously curated MIMIC-IV custom feature set optimized for IHM prediction, constructed through a reproducible preprocessing pipeline and feature selection strategy.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Plasmonic Core–Shell–Satellites with Abundant Electromagnetic Hotspots for Highly Sensitive and Reproducible SERS Detection

In this work, we develop a Ag@Al2O3@Ag plasmonic core–shell–satellite (PCSS) to achieve highly sensitive and reproducible surface-enhanced Raman spectroscopy (SERS) detection of probe molecules. To fabricate PCSS nanostructures, we employ a simple hierarchical dewetting process of Ag films coupled with an atomic layer deposition (ALD) method for the Al2O3 shell. Compared to bare Ag nanoparticles, several advantages of fabricating PCSS nanostructures are discovered, including high surface roughness, high density of nanogaps between Ag core and Ag satellites, and nanogaps between adjacent Ag satellites. Finite-difference time-domain (FDTD) simulations of the PCSS nanostructure confirm an enhancement in the electromagnetic field intensity (hotspots) in the nanogap between the Ag core and the satellite generated by the Al2O3 shell, due to the strong core–satellite plasmonic coupling. The as-prepared PCSS-based SERS substrate demonstrates an enhancement factor (EF) of 1.7 × 107 and relative standard deviation (RSD) of ~7%, endowing our SERS platform with highly sensitive and reproducible detection of R6G molecules. We think that this method provides a simple approach for the fabrication of PCSS by a solid-state technique and a basis for developing a highly SERS-active substrate for practical applications.

47 OTHER INSTRUMENTATION↗

DOE OSTI2020

The Gut Microbiome and Alcoholic Liver Disease: Ethanol Consumption Drives Consistent and Reproducible Alteration in Gut Microbiota in Mice

Phenotypic health effects, both positive and negative, have been well studied in association with the consumption of alcohol in humans as well as several other mammals including mice. Many studies have also associated these same health effects and phenotypes to specific members of gut microbiome communities. Here we utilized a chronic plus binge ethanol feed model (Gao-binge model) to explore microbiome community changes across three independent experiments performed in mice. We found significant and reproducible differences in microbiome community assemblies between ethanol-treated mice and control mice on the same diet absent of ethanol. We also identified significant differences in gut microbiota occurring temporally with ethanol treatment. Peak shift in communities was observed 4 days after the start of daily alcohol consumption. We quantitatively identified many of the bacterial genera indicative of these ethanol-induced shifts including 20 significant genera when comparing ethanol treatments with controls and 14 significant genera based on temporal investigation. Including overlap of treatment with temporal shifts, we identified 25 specific genera of interest in ethanol treatment microbiome shifts. Shifts coincide with observed presentation of fatty deposits in the liver tissue, i.e., Alcoholic Liver Disease-associated phenotype. The evidence presented herein, derived from three independent experiments, points to the existence of a common, reproducible, and characterizable “mouse ethanol gut microbiome”.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2026

Enhanced climate reproducibility testing with false discovery rate correction

Simulating the Earth's climate is an important and complex problem, thus climate models are similarly complex, comprised of millions of lines of code. In order to appropriately utilize the latest computational and software infrastructure advancements in Earth system models running on modern hybrid computing architectures to improve their performance, precision, accuracy, or all three; it is important to ensure that model simulations are repeatable and robust. This introduces the need for establishing statistical or non-bit-for-bit reproducibility, since bit-for-bit reproducibility may not always be achievable. Here, we propose a short-simulation ensemble-based test for an atmosphere model to evaluate the null hypothesis that modified model results are statistically equivalent to that of the original model. We implement this test in version 2 of the US Department of Energy's Energy Exascale Earth System Model (E3SM). The test evaluates a standard set of output variables across the two simulation ensembles and uses a false discovery rate correction to account for multiple testing. The false positive rates of the test are examined using re-sampling techniques on large simulation ensembles and are found to be lower than the currently implemented bootstrapping-based testing approach in E3SM. We also evaluate the statistical power of the test using perturbed simulation ensemble suites, each with a progressively larger magnitude of change to a tuning parameter. The new test is generally found to exhibit more statistical power than the current approach, being able to detect smaller changes in parameter values with higher confidence.

Kelleher, Michael E. [Oak Ridge National Laborator↗

NASA NTRS1970

Multispectral facsimile reproducer

Facsimile reproducer records spatially well-registered true-color and false-color video data on tape. Proposed optical arrangement allows light sources to be adjusted along the mirror scanning direction in such a manner that no electronic delays are required if the reproducer is operated synchronously with the camera.

Burcher, E. E.↗

NASA NTRS1988

Reproducible analyses of microbial food for advanced life support systems

The use of yeasts in controlled ecological life support systems (CELSS) for microbial food regeneration in space required the accurate and reproducible analysis of intracellular carbohydrate and protein levels. The reproducible analysis of glycogen was a key element in estimating overall content of edibles in candidate yeast strains. Typical analytical methods for estimating glycogen in Saccharomyces were not found to be entirely aplicable to other candidate strains. Rigorous cell lysis coupled with acid/base fractionation followed by specific enzymatic glycogen analyses were required to obtain accurate results in two strains of Candida. A profile of edible fractions of these strains was then determined. The suitability of yeasts as food sources in CELSS food production processes is discussed.

Petersen, Gene R.↗

NASA NTRS1997

Visual-Stratigraphic Dating of the GISP2 Ice Core: Basis, Reproducibility, and Application

Annual layers are visible in the Greenland Ice Sheet Project 2 ice core from central Greenland, allowing rapid dating of the core. Changes in bubble and grain structure caused by near-surface, primarily summertime formation of hoar complexes provide the main visible annual marker in the Holocene, and changes in "cloudiness" of the ice correlated with dustiness mark Wisconsinan annual cycles; both markers are evident and have been intercalibrated in early Holocene ice. Layer counts are reproducible between different workers and for one worker at different times, with 1% error over century-length times in the Holocene. Reproducibility is typically 5% in Wisconsinan ice-age ice and decreases with increasing age and depth. Cumulative ages from visible stratigraphy are not significantly different from independent ages of prominent events for ice older than the historical record and younger than approximately 50,000 years. Visible observations are not greatly degraded by "brittle ice" or many other core-quality problems, allowing construction of long, consistently sampled time series. High accuracy requires careful study of the core by dedicated observers.

Alley, R. B.↗

NASA NTRS2002

Reproducibility and Repeatability of Tensile and Low-Cycle Fatigue Properties in Propulsion Grade Hydrogen

Hydrogen has the potential of increased use in the future as an environmentally friendly fuel. It has, however, shown a tendency to embrittle some materials. To be used in a safe manner and to exploit its full potential, it will be necessary to develop a database of material properties in hydrogen environment. The tests needed to produce this data are costly to perform (tensile test cost 25 times more and low cycle fatigue test are 55 times as expensive). Moreover, there is presently a lack of universal test methods to ensure standardized data within the hydrogen community. Each of the industries that work with hydrogen (aerospace, petroleum, fuel cells, etc.) performs tests by their own laboratory-developed methods, thus rendering cross- comparisons of material property data highly questionable. It is extremely important that data generated in a hydrogen environment be done to a standard that reduces variance to a minimum and allows direct comparison of test results from different laboratories. Doing so will assure that all data generated can be used to further our understanding of the hydrogen effects and to make sure components/products designed for hydrogen are the safest and most reliable possible. This paper reviews the results of two 'round-robin' programs conducted by NASA-MSFC. These two programs examined the reproducibility and repeatability of tensile and low-cycle fatigue test results in high-pressure hydrogen environments. The studies indicated that even with the tightest controls available from current commercial standards, the reproducibility (between different laboratories) and repeatability (within a laboratory) results of the tensile tests exhibited five times the variance as in standard ambient air tests. The variance with the LCF tests were on the same order as with air tests, but that was due to the large variation present in the last Interlaboratory air program. The paper concludes with a recommendation for a program that would allow the development of improved test methods, leading to lower variance in the generation of mechanical property data in the future.

Vesely, E. J.↗