Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Reproducibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

The growth in scientific data volumes has resulted in a need to scale up processing and analysis pipelines using High Performance Computing (HPC) systems. These workflows need interactive, reproducible analytics at scale. The Jupyter platform provides core capabilities for interactivity but was not designed for HPC systems. In this paper, we outline our efforts that bring together core technologies based on the Jupyter Platform to create interactive, reproducible analytics at scale on HPC systems. Our work is grounded in a real world science use case-applying geophysical simulations and inversions for imaging the subsurface. Our core platform addresses three key areas of the scientific analysis workflow-reproducibility, scalability, and interactivity. We describe our implemention of a system, using Binder, Science Capsule, and Dask software. We demonstrate the use of this software to run our use case and interactively visualize real-Time streams of HDF5 data.

containers↗
DOE OSTI2019

Computational Reproducibility of Scientific Workflows at Extreme Scales

We propose an approach for improved reproducibility that includes capturing and relating provenance characteristics and performance metrics, in a hybrid queriable system, the ProvEn server. The system capabilities are illustrated on two use cases: scientific reproducibility of results in the ACME climate simulations and performance reproducibility in molecular dynamics workflows on HPC computing platforms.

provenance, Reproducibility, Climate & Earth Syste↗
DOE OSTI2019

Reliability of the {sup 123}I-mIBG heart/mediastinum ratio: Results of a multicenter test–retest reproducibility study

A quantitative measurement, the Heart-to-Mediastinum (H/M) ratio of counts derived from a planar acquisition approximately 4 hours after injection of {sup 123}I-mIBG, is a strong predictor of outcomes in patients with stable class II-III heart failure and LVEF ≤ 35%. This study assessed the test–retest reproducibility of the H/M ratio in such patients. 47 subjects with class II-III systolic heart failure and LVEF ≤ 35% were tested at two time intervals separated by 5 to 14 days. Subjects were imaged twice on the same camera using the same radionuclide dose. Images were sent to a core analysis lab, where three nuclear technologists independently determined the H/M ratios. The primary endpoint was test–retest H/M ratio reproducibility calculated as the absolute difference in mean value determined by the three readers. Mean subject age was 65 ± 12 years, 85% were male, and mean BMI was 29 ± 6 kg/m{sup 2}. Mean injected activity was 10.18 ± 0.43 mCi for first dose and 10.09 ± 0.52 mCi for the second dose. The mean and SD values for first and repeat studies were almost identical: the 95% confidence interval of the mean test–retest difference was 0.055 to 0.076. Bland–Altman plots showed no systematic effect of the H/M ratio on the magnitude of the difference between replicate measurements. Inter-reader measurements were nearly identical. There were no serious adverse events despite exposure to {sup 123}I-mIBG on 2 occasions in a short time period. The Heart-to-Mediastinum ratio of {sup 123}I-mIBG is a consistent and highly reproducible measurement in stable Class II to III heart failure patients.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2026

From Reproducible Edge–Cloud Experimentation to Real-World Practice: The E2Clab Experience

Reproducibility is already difficult in distributed systems; on the computing continuum, it becomes substantially harder. Applications that span sensing devices, edge and fog resources, and cloud platforms must be evaluated across heterogeneous hardware, variable network conditions, cross-layer orchestration decisions, and long-running workflow lifecycles. We use E2Clab as a case study to examine these challenges and their implications for experimental methodology. We explain why reproducible experimentation is harder on the continuum, then revisit E2Clab as an initial response based on explicit modeling of infrastructure, workflow lifecycle, and artifacts. Lastly, we discuss how its evolution toward more realistic application settings can be understood through the lens of Translational Computer Science. We argue that reproducible continuum experimentation requires methods that are rigorous enough for research while remaining adaptable to real-world practice.

42 ENGINEERING↗
DOE OSTI2023

Towards Redefining the Reproducibility in Quantum Computing: A Data Analysis Approach on NISQ Devices

Although the building of quantum computers has kept making rapid progress in recent years, noise is still the main challenge for any application to leverage the power of quantum computing. Existing works addressing noise in quantum devices proposed noise reduction when deploying a quantum algorithm to a specified quantum computer. The reproducibility issue of quantum algorithms has been raised since the noise levels vary on different quantum computers. Importantly, existing works largely ignore the fact that the noise of quantum devices varies as time goes by. Therefore, reproducing the results on the same hardware will even become a problem. We analyze the reproducibility of quantum machine learning (QML) algorithms based on daily model training and execution data collection. Our analysis shows a correlation between our QML models’ test accuracy and quantum computer hardware’s calibration features. We also demonstrate that noisy simulators for quantum computers are not a reliable tool for quantum machine learning applications.

Senapati, Priyabrata↗
DOE OSTI2024

Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications

Run to run variability in parallel programs caused by floating-point non-associativity has been known to significantly affect reproducibility in iterative algorithms, due to accumulating errors. Non-reproducibility can critically affect the efficiency and effectiveness of correctness testing for stochastic programs. Recently, the sensitivity of deep learning training and inference pipelines to floating-point non-associativity has been found to sometimes be extreme. It can prevent certification for commercial applications, accurate assessment of robustness and sensitivity, and bug detection. New approaches in scientific computing applications have coupled deep learning models with high-performance computing, leading to an aggravation of debugging and testing challenges. Here we perform an investigation of the statistical properties of floating-point non-associativity within modern parallel programming models, and analyze performance and productivity impacts of replacing atomic operations with deterministic alternatives on GPUs. We examine the recently-added deterministic options in PyTorch within the context of GPU deployment for deep learning, uncovering and quantifying the impacts of input parameters triggering run to run variability and reporting on the reliability and completeness of the documentation. Finally, we evaluate the strategy of exploiting automatic determinism that could be provided by deterministic hardware, using the Groq LPUTM accelerator for inference portions of the deep learning pipeline. We demonstrate the benefits that a hardware-based strategy can provide within reproducibility and correctness efforts.

Shanmugavelu, Sanjif↗
DOE OSTI2020

Consistency and reproducibility in atomic layer deposition

Atomic layer deposition (ALD) is a thin film synthesis technique that can provide exquisite accuracy and precision in film thickness and composition even on complex, large area substrates. Based on self-limiting surface chemistry, ALD can be insensitive to process conditions and reactor designs, allowing an ALD process developed in one lab to be easily reproduced in other labs. In practice, however, ALD is sometimes difficult to reproduce or replicate, and the results can vary substantially between ALD reactors and between labs. This is exemplified by large deviations in reports on the growth of, e.g., Al2O3, FeOx, and TiO2 given the same precursors under similar conditions. Furthermore, the problem of irreproducibility seems to be growing as ALD is adopted by more researchers and integrated into new applications. In this article, the authors highlight some of the major sources of variations and errors and common misconceptions related to ALD. In particular, the authors focus on issues related to precursors, substrates, and deposition tools. The authors illustrate these problems through examples from the literature, and they present results from numerical simulations that describe how nonidealities would manifest in thickness profiles in a typical cross-flow reactor. They also describe how reproducibility in ALD is linked to consistent experimental practice and reporting between labs. The authors’ hope is that by educating newcomers to ALD and advocating for consistent reporting of deposition conditions, they can minimize irreproducibility and enable ALD practitioners to realize the full potential afforded by self-limiting surface chemistry.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Breaking the reproducibility barrier with standardized protocols for plant–microbiome research

Inter-laboratory replicability is crucial yet challenging in microbiome research. Leveraging microbiomes to promote soil health and plant growth requires understanding underlying molecular mechanisms using reproducible experimental systems. In a global collaborative effort involving five laboratories, we aimed to help advance reproducibility in microbiome studies by testing our ability to replicate synthetic community assembly experiments. Our study compared fabricated ecosystems constructed using two different synthetic bacterial communities, the model grass Brachypodium distachyon, and sterile EcoFAB 2.0 devices. All participating laboratories observed consistent inoculum-dependent changes in plant phenotype, root exudate composition, and final bacterial community structure, where Paraburkholderia sp. OAS925 could dramatically shift microbiome composition. Comparative genomics and exudate utilization linked the pH-dependent colonization ability of Paraburkholderia, which was further confirmed with motility assays. The study provides detailed protocols, benchmarking datasets, and best practices to help advance replicable science and inform future multi-laboratory reproducibility studies.

Novak, Vlastimil↗
DOE OSTI2020

Reproducibility and Optimization Techniques for Evaluated Data in the Resolved Resonance Region [Slides]

This presentation discusses the experimental setup input parameters that are basic quantities for reproducibility. The goal is to increase quality of the evaluated data and decrease time needed for an evaluation. Full reproducibility should be achieved efficiently by including any type of constraints in the optimization procedure and to improve computational time to reach convergence. The goal of an automated reproducibility aligns to the goal of “certified” evaluated data, i.e., defined by a well-defined metric.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
NASA NTRS1978

Reproducibility of structural strength and stiffness for graphite-epoxy aircraft spoilers

Structural strength reproducibility of graphite epoxy composite spoilers for the Boeing 737 aircraft was evaluated by statically loading fifteen spoilers to failure at conditions simulating aerodynamic loads. Spoiler strength and stiffness data were statistically modeled using a two parameter Weibull distribution function. Shape parameter values calculated for the composite spoiler strength and stiffness were within the range of corresponding shape parameter values calculated for material property data of composite laminates. This agreement showed that reproducibility of full scale component structural properties was within the reproducibility range of data from material property tests.

Howell, W. E.↗
NASA NTRS1995

The effect of sample holder material on ion mobility spectrometry reproducibility

When a positive detection of a narcotic occurs during the search of a vessel, a decision has to be made whether further intensive search is warranted. This decision is based in part on the results of a second sample collected from the same area. Therefore, the reproducibility of both sampling and instrumental analysis is critical in terms of justifying an in depth search. As reported at the 2nd Annual IMS Conference in Quebec City, the U.S. Coast Guard has determined that when paper is utilized as the sample desorption medium for the Barringer IONSCAN, the analytical results using standard reference samples are reproducible. A study was conducted utilizing papers of varying pore sizes and comparing their performance as a desorption material relative to the standard Barringer 50 micron Teflon. Nominal pore sizes ranged from 30 microns down to 2 microns. Results indicate that there is some peak instability in the first two to three windows during the analysis. The severity of the instability was observed to increase as the pore size of the paper is decreased. However, the observed peak instability does not create a situation that results in a decreased reliability or reproducibility in the analytical result.

Jadamec, J. Richard↗
NASA NTRS1999

Autonomous Agents: The Origins and Co-Evolution of Reproducing Molecular Systems

The central aim of this award concerned an investigation into, and adequate formulation of, the concept of an "autonomous agent." If we consider a bacterium swimming upstream in a glucose gradient, we are willing to say of the bacterium that it is going to get food. That is, we are willing, and do, describe the bacterium as acting on its own behalf in an environment. All free living cells are, in this sense, autonomous agents. But the bacterium is "just" a set of molecules. We define an autonomous agent as a physical system able to act on its own behalf in an environment, then ask, "What must a physical system be to be an autonomous agent?" The tentative definition for a molecular autonomous agent is that it must be self-reproducing and carry out at least one thermodynamic work cycle. The work carried out in this grant involved, among other features, the development of a detailed model of a molecular autonomous agent, and study of the kinetics of this system. In particular, a molecular autonomous agent must, by the above tentative definition, not only reproduce, but must carry out at least one work cycle. I took, as a simple example of a self-reproducing molecular system, the single-stranded DNA hexamer 3'CCGCGG5' which can line up and ligate its two complementary trimers, 5'CCG3' and 5'CGG3'. But the two ligated trimers constitute the same molecular sequence in the 3' to 5' direction as the initial hexamer, hence this system is autocatalytic. On the other hand the above system is not yet an autonomous agent. At the minimum, autonomous agents, as I have defined them, are a new class of chemical reaction network. At a maximum, they may constitute a proper definition of life itself.

Kauffman, Stuart↗
NASA NTRS2004

Measurement of Force Balance Repeatability and Reproducibility in the NTF

A recently published statistical approach for measuring and evaluating wind tunnel force balance repeatability and reproducibility is applied to three check standard tests in the National Transonic Facility at NASA Langley Research Center. Two different airframe models and force balances were used. The short-term repeatability and within-test reproducibility are separately estimated and correlations with tunnel parameters are carried out. Conjectures are presented for the development of scaling laws for predicting the repeatability and reproducibility of other force balance tests in the tunnel.

Hemsch, M. J.↗
NASA NTRS2010

Repeatability and Reproducibility of Compression Strength Measurements Conducted According to ASTM E9

Ten commercial laboratories participated in an interlaboratory study to establish the repeatability and reproducibility of compression strength tests conducted according to ASTM International Standard Test Method E9. The test employed a cylindrical aluminum AA2024-T351 test specimen. Participants measured elastic modulus and 0.2 % offset yield strength, YS(0.2 % offset), using an extensometer attached to the specimen. The repeatability and reproducibility of the yield strength measurement, expressed as coefficient of variations were cv(sub r)= 0.011 and cv(sub R)= 0.020 The reproducibility of the test across the laboratories was among the best that has been reported for uniaxial tests. The reported data indicated that using diametrically opposed extensometers, instead of a single extensometer doubled the precision of the test method. Laboratories that did not lubricate the ends of the specimen measured yield stresses and elastic moduli that were smaller than those measured in laboratories that lubricated the specimen ends. A finite element analysis of the test specimen deformation for frictionless and perfect friction could not explain the discrepancy, however. The modulus measured from stress-strain data were reanalyzed using a technique that finds the optimal fit range, and applies several quality checks to the data. The error in modulus measurements from stress-strain curves generally increased as the fit range decreased to less than 40 % of the stress range.

Luecke, William E.↗
DOE OSTI2021

A reproducibility study of "Augmenting Genetic Algorithms with Deep Neural Networks for Exploring the Chemical Space"

Nigam et al. reported a genetic algorithm (GA) utilizing the SELFIES representation and also propose an adaptive, neural network-based penalty that is supposed to improve the diversity of the generated molecules. The main claims of the paper are that this GA outperforms other generative techniques (as measured by the penalized logP) and that a neural network-based adaptive penalty increases the diversity of the generated molecules. In this work, we investigated the reproducibility of their claims. Overall, we were able to reproduce comparable results using the SELFIES-based GA, but mostly by exploiting deficiencies of the (easily optimizable) fitness function (i.e., generating long, sulfur containing chains). In addition, we reproduce results showing that the discriminator can be used to bias the generation of molecules to ones that are similar to the reference set. Lastly, we attempted to quantify the evolution of the diversity, understand the influence of some hyperparameters, and propose improvements to the adaptive penalty.

Jablonka, Kevin Maik↗
DOE OSTI2024

Image-based modeling of coupled electro-chemo-mechanical behavior of Li-ion battery cathode using an interface-modified reproducing kernel particle method

Abstract An interface-modified reproducing kernel particle method (IM-RKPM) is introduced in this work to allow for a direct model construction from image pixels of heterogeneous polycrystalline Li-ion battery microstructures. The interface-modified reproducing kernel (IM-RK) approximation is constructed through scaling of a kernel function by a regularized distance function in conjunction with strategic placement of interface node locations. This leads to RK shape functions with either weak or strong discontinuities across material interfaces, suitable for modeling various interface mechanics. With the placement of a triple junction node and distance-based scaling of kernel functions, the resulting IM-RK shape function also possesses proper discontinuities at the triple junctions. This IM-RK approximation effectively remedies the well-known Gibb’s oscillation in the smooth approximation of discontinuities. Different from the conventional meshfree approaches for interface discontinuities, this IM-RK approach is done without additional degrees of freedom associated with the enrichment functions, and it is formulated with the standard procedures in the RK shape function construction. This work focuses on identifying the accuracy and convergence properties of IM-RKPM for modeling the coupled electro-chemo-mechanical system. A linear patch test is formulated and numerically tested for the electro-chemo-mechanical coupled problem with a Butler–Volmer boundary condition representing the physical conditions in Li-ion battery microstructures. This is followed by verification of the optimal rates of convergence of IM-RKPM for solving the coupled problem with higher order solutions. The image-based modeling of Li-ion battery microstructures in the numerical examples demonstrates the applicability of the proposed method to realistic Li-ion battery materials modeling.

25 ENERGY STORAGE↗
DOE OSTI2024

A quasi-meshfree method for constructing boundary-aware reproducing bases on geometrically complex domains using manifold geodesics

In many applications, physical domains are geometrically complex making it challenging to perform coarse-scale approximation. A defeaturing process is often used to simplify the domain in preparation for approximation and analysis at the coarse scale. In this paper, a methodology is presented for constructing a coarse-scale reproducing basis on geometrically complex domains given an initial fine-scale mesh of the fully featured domain. The initial fine-scale mesh can be of poor quality and extremely refined. The construction of the basis functions begins with a coarse-scale covering of the domain and generation of weighting functions with local support. Manifold geodesics are used to define distances within the local support for general applicability to non-convex domains. Conventional moving least squares is used to construct the coarse-scale reproducing basis. Applications in quasi-interpolation and linear elasticity are presented.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

CUF-Links: Continuous and Ubiquitous FAIRness Linkages for Reproducible Research

Despite much creative work on methods and tools, reproducibility— the ability to repeat the computational steps used to obtain a research result—remains elusive. One reason for these difficulties is that extant tools for capturing research processes, while powerful, often fail to capture vital connections as research projects grow in extent and complexity. We explain here how these interstitial connections can be preserved via simple methods that integrate easily with current work practices to capture basic information about every data product consumed or produced in a project. By thus extending the scope of findable, accessible, interoperable, and reusable (FAIR) data in both time and space to enable the creation of a continuous chain of Continuous and Ubiquitous FAIRness linkages (CUFlinks) from inputs to outputs, such mechanisms can facilitate capture of the provenance linkages that are essential to reproducible research. Herein we give examples of mechanisms that can facilitate the use of these methods, and review how they have been applied in practice.

97 MATHEMATICS AND COMPUTING↗