Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Reproducibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Reproducible Data Processing Research for the CABRI R.I.A. experiments Acoustic Emission signal analysis

The CABRI facility is an experimental nuclear reactor of the French Atomic Energy Commission (CEA) designed to study the behaviour of fuel rods at high burnup under Reactivity Initiated Accident (R.I.A.) conditions such as the scenario of a control rod ejection. During the experimental phase, the behaviour of the fuel element generates acoustic waves which can be detected by two microphones placed upstream and downstream from the test device. Studies carried out on the last fourteen tests showed the interest in carrying out temporal and spectral analyses on these signals by showing the existence of signatures which can be correlated with physical phenomena. We want presently to return to this rich data in order to have a new point of view by applying modern signal processing methods. Such an antecedent works resumption leads to some difficulties. Although all the raw data are accessible in the form of text files, analyses and graphics representations were not clear in reproducing from the former studies since the people who were in charge of the original work have left the laboratory and it is not easy when time passes, even with our own work, to be able to remember the steps of data manipulations and the exact setup. Thus we decided to consolidate the availability of the data and its manipulation in order to provide a robust data processing workflow to the experimentalists before doing any further investigations. To tackle this issue of strong links between data, treatments and the generation of documents, we adopted a Reproducible Research paradigm. We shall first present the tools chosen in our laboratory to implement this workflow and, then we shall describe the global perception carried out to continue the study of the Acoustic Emission signals recorded by the two microphones during the last fourteen CABRI R.I.A. tests. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Building the I (Interoperability) of FAIR for performance reproducibility of large-scale composable workflows in RECUP

Abstract-Scientific computing communities increasingly run their experiments using complex data- and compute-intensive workflows that utilize distributed and heterogeneous architectures targeting numerical simulations and machine learning, often executed on the Department of Energy Leadership Computing Facilities (LCFs). We argue that a principled, systematic approach to implementing FAIR principles at scale, including fine-grained metadata extraction and organization, can help with the numerous challenges to performance reproducibility posed by such workflows. We extract workflow patterns, propose a set of tools to manage the entire life cycle of performance metadata, and aggregate them in an HPC-ready framework for reproducibility (RECUP). We describe the challenges in making these tools interoperable, preliminary work, and lessons learned from this experiment.

97 MATHEMATICS AND COMPUTING↗

Managing Software Provenance to Enhance Reproducibility in Computational Research

Scientific processes rely on software as an important tool for data acquisition, analysis, and discovery. Over the years, sustainable software development practices have made progress in being considered as an integral component of research. However, management of computation-based scientific studies is often left to individual researchers who design their computational experiments based on personal preferences and the nature of the study. Here, we believe that the quality, efficiency, and reproducibility of computation-based scientific research can be improved by explicitly creating an execution environment that allows researchers to provide a clear record of traceability. This is particularly relevant to complex computational studies in high-performance computing (HPC) environments. In this article, we review the documentation required to maintain a comprehensive record of HPC computational experiments for reproducibility. We also provide an overview of tools and practices that we have developed to perform such studies around Flash-X, a multiphysics scientific software.

97 MATHEMATICS AND COMPUTING↗

QAOAKit: A Toolkit for Reproducible Study, Application, and Verification of the QAOA

Understanding the best known parameters, performance, and systematic behavior of the Quantum Approximate Optimization Algorithm (QAOA) remain open research questions, even as the algorithm gains popularity. We introduce QAOAKit, a Python toolkit for the QAOA built for exploratory research. QAOAKit is a unified repository of preoptimized QAOA parameters and circuit generators for common quantum simulation frameworks. We combine, standardize, and cross-validate previously known parameters for the MaxCut problem, and incorporate this into QAOAKit. We also build conversion tools to use these parameters as inputs in several quantum simulation frameworks that can be used to reproduce, compare, and extend known results from various sources in the literature. We describe QAOAKit and provide examples of how it can be used to reproduce research results and tackle open problems in quantum optimization.

Shaydulin, Ruslan↗

A Python Instrument Control and Data Acquisition Suite for Reproducible Research

Tools that standardize and automate experimental data collection are needed for greater confidence in research results. The National Synchrotron Light Source-II (NSLS-II) has generated an open-source Python data acquisition, management, and analysis software suite that automates X-ray experiments and collects an experimental record that facilitates complete reproducibility. Here, we show that the NSLS-II tools are not only useful for X-ray science at large-scale facilities by presenting an add-on package that adapts these tools for use in a small laboratory with common physics and electrical engineering instruments. The composite software suite eases and automates the execution of experiments, records extensive metadata, stores data in portable containers, and speeds up the analysis through tools for comprehensive searches. In total, this software suite increases the reproducibility of laboratory experiments. We demonstrate the software via the evaluation of two lock-in amplifiers-the miniature ADA2200 and the ubiquitous Stanford Research Systems (SRS) SR810. The frequency resolution, signal-to-noise ratio, and dynamic reserve of the lock-in amplifiers are measured and presented. The usage of the software suite is described throughout these measurements so that the reader can implement the tools in their lab.

97 MATHEMATICS AND COMPUTING↗

BASTet: Shareable and Reproducible Analysis and Visualization of Mass Spectrometry Imaging Data via OpenMSI

Mass spectrometry imaging (MSI) is a transformative imaging method that supports the untargeted, quantitative measurement of the chemical composition and spatial heterogeneity of complex samples with broad applications in life sciences, bioenergy, and health. While MSI data can be routinely collected, its broad application is currently limited by the lack of easily accessible analysis methods that can process data of the size, volume, diversity, and complexity generated by MSI experiments. The development and application of cutting-edge analytical methods is a core driver in MSI research for new scientific discoveries, medical diagnostics, and commercial-innovation. However, the lack of means to share, apply, and reproduce analyses hinders the broad application, validation, and use of novel MSI analysis methods. To address this central challenge, we introduce the Berkeley Analysis and Storage Toolkit (BASTet), a novel framework for shareable and reproducible data analysis that supports standardized data and analysis interfaces, integrated data storage, data provenance, workflow management, and a broad set of integrated tools. Based on BASTet, we describe the extension of the OpenMSI mass spectrometry imaging science gateway to enable web-based sharing, reuse, analysis, and visualization of data analyses and derived data products. Here, we demonstrate the application of BASTet and OpenMSI in practice to identify and compare characteristic substructures in the mouse brain based on their chemical composition measured via MSI.

97 MATHEMATICS AND COMPUTING↗

Science Capsule: Towards Sharing and Reproducibility of Scientific Workflows

Workflows are increasingly processing large volumes of data from scientific instruments, experiments and sensors. These workflows often consist of complex data processing and analysis steps that might include a diverse ecosystem of tools and also often involve human-in-the-loop steps. Sharing and reproducing these workflows with collaborators and the larger community is critical but hard to do without the entire context of the workflow including user notes and execution environment. In this paper, we describe Science Capsule, which is a framework to capture, share, and reproduce scientific workflows. Science Capsule captures, manages and represents both computational and human elements of a workflow. It automatically captures and processes events associated with the execution and data life cycle of workflows, and lets users add other types and forms of scientific artifacts. Science Capsule also allows users to create `workflow snapshots' that keep track of the different versions of a workflow and their lineage, allowing scientists to incrementally share and extend workflows between users. Our results show that Science Capsule is capable of processing and organizing events in near real-time for high-throughput experimental and data analysis workflows without incurring any significant performance overheads.

Ghoshal, Devarshi↗

Evolvix BEST Names for semantic reproducibility across code2brain interfaces

Names in programming are vital for understanding the meaning of code and big data. We define code2brain (C2B) interfaces as maps in compilers and brains between meaning and naming syntax, which help to understand executable code. While working toward an Evolvix syntax for general‐purpose programming that makes accurate modeling easy for biologists, we observed how names affect C2B quality. To protect learning and coding investments, C2B interfaces require long‐term backward compatibility and semantic reproducibility (accurate reproduction of computational meaning from coder‐brains to reader‐brains by code alone). Semantic reproducibility is often assumed until confusing synonyms degrade modeling in biology to deciphering exercises. We highlight empirical naming priorities from diverse individuals and roles of names in different modes of computing to show how naming easily becomes impossibly difficult. We present the Evolvix BEST (Brief, Explicit, Summarizing, Technical) Names concept for reducing naming priority conflicts, test it on a real challenge by naming subfolders for the Project Organization Stabilizing Tool system, and provide naming questionnaires designed to facilitate C2B debugging by improving names used as keywords in a stabilizing programming language. Our experiences inspired us to develop Evolvix using a flipped programming language design approach with some unexpected features and BEST Names at its core.

97 MATHEMATICS AND COMPUTING↗

Materials characterization: Can artificial intelligence be used to address reproducibility challenges?

Material characterization techniques are widely used to characterize the physical and chemical properties of materials at the nanoscale and, thus, play central roles in material scientific discoveries. However, the large and complex datasets generated by these techniques often require significant human effort to interpret and extract meaningful physicochemical insights. Artificial intelligence (AI) techniques such as machine learning (ML) have the potential to improve the efficiency and accuracy of surface analysis by automating data analysis and interpretation. In this perspective paper, we review the current role of AI in surface analysis and discuss its future potential to accelerate discoveries in surface science, materials science, and interface science. We highlight several applications where AI has already been used to analyze surface analysis data, including the identification of crystal structures from XRD data, analysis of XPS spectra for surface composition, and the interpretation of TEM and SEM images for particle morphology and size. We also discuss the challenges and opportunities associated with the integration of AI into surface analysis workflows. These include the need for large and diverse datasets for training ML models, the importance of feature selection and representation, and the potential for ML to enable new insights and discoveries by identifying patterns and relationships in complex datasets. Most importantly, AI analyzed data must not just find the best mathematical description of the data, but it must find the most physical and chemically meaningful results. In addition, the need for reproducibility in scientific research has become increasingly important in recent years. The advancement of AI, including both conventional and the increasing popular deep learning, is showing promise in addressing those challenges by enabling the execution and verification of scientific progress. By training models on large experimental datasets and providing automated analysis and data interpretation, AI can help to ensure that scientific results are reproducible and reliable. Although integration of knowledge and AI models must be considered for the transparency and interpretability of models, the incorporation of AI into the data collection and processing workflow will significantly enhance the efficiency and accuracy of various surface analysis techniques and deepen our understanding at an accelerated pace.

Materials Science↗

SU-E-I-91: Reproducibility in Prescribed Dose in AEC CT Scans Due to Table Height, Patient Size, and Localizer Acquisition Order

Purpose: In CT scanners, the automatic exposure control (AEC) tube current prescription depends on the acquired prescan localizer image(s). The purpose of this study was to quantify the effect that table height, patient size, and localizer acquisition order may have on the reproducibility in prescribed dose. Methods: Three phantoms were used for this study: the Mercury Phantom (comprises three tapered and four uniform regions of polyethylene 16, 23, 30, and 37 cm in diameter), acrylic sheets, and an adult anthropomorphic phantom. Phantoms were positioned per clinical protocol by our chief CT technologist or broader symmetry. Using a GE Discovery CT750HD scanner, a lateral (LAT) and posterior-anterior (PA) localizer was acquired for each phantom at different table heights. AEC scan acquisitions were prescribed for each combination of phantom, localizer orientation, and table height; the displayed volume CTDI was recorded for each. Results were analyzed versus table height. Results: For the two largest Mercury Phantom section scans based on the PA localizer, the percent change in volume CTDI from ideal were at least 20% lower and 35% greater for table heights 4 cm above and 4 cm below proper centering, respectively. For scans based on the LAT localizer, the percent change in volume CTDI from ideal were no greater than 12% different for 4 cm differences in table height. The properly centered PA and LAT localizer-based volume CTDI values were within 13% of each other. Conclusion: Since uncertainty in vertical patient positioning is inherently greater than lateral positioning and because the variability in dose exceeds any dose penalties incurred, the LAT localizer should be used to precisely and reproducibly deliver the intended amount of radiation prescribed by CT protocols. CT protocols can be adjusted to minimize the expected change in average patient dose.

60 APPLIED LIFE SCIENCES↗

TU-C-12A-07: Characterization of Longitudinal Reproducibility of Quantitative Diffusion Imaging Data Acquired with Four Different Protocols Using a Phantom

Purpose: To characterize and compare the longitudinal reproducibility of diffusion imaging data acquired with four different protocols using a phantom. Methods: The Diffusive Quantitative Imaging Phantom (DQIP) was constructed using fifteen cylindrical compartments within a larger compartment, filled with deionized water doped with CuSO4 and NaCl. The smaller compartments contained arrays of hexagonal or cylindrical glass capillaries of varying inner diameters, for differing restraint of water diffusion. The sensitivity of diffusion imaging metrics to signal-to-noise ratio (SNR) was probed by doping compartments with differing ratios of deuterium oxide to H2O. A cork phantom enclosure was constructed to increase thermal stability during scanning and a cork holder was made to reproduce scanner positioning. Four different protocols of DWI (diffusion weighted imaging) and DTI (Diffusion tensor imaging) imaging were assembled on a GE Excite HDx 3.0T MRI scanner to collect imaging data over 9-10 days. Data was processed with in-house software created in Matlab to obtain fractional anisotropy (FA) and apparent diffusion coefficient (ADC) values. Results: All DTI and DWI sequences showed good longitudinal stability of mean FA and ADC values per compartment, exhibiting low standard deviation ∼9%. A t-test was performed to compare mean FA values from the DTI clinical protocol to those of the DTI special protocol, indicating significantly different values in the majority of compartments. ANOVA performed on ADC values for all DTI and DWI sequences also showed significantly different values in a majority of compartments. Conclusion: This work has the potential for quantifying systemic variations between diffusion imaging sequences from different platforms. Characterization of DWI and DTI performance were done over four sequences with predictable results. This data suggests that the DQIP phantom may be a reliable method of monitoring day-to-day and scan-to-scan variation in diffusion imaging sequences from different platforms. Schott Glass North America and The Phantom Laboratory have donated materials and personnel time to this project.

60 APPLIED LIFE SCIENCES↗

SU-E-J-252: Reproducibility of Radiogenomic Image Features: Comparison of Two Semi-Automated Segmentation Methods

Purpose: Objective and reliable quantification of imaging phenotype is an essential part of radiogenomic studies. We compared the reproducibility of two semi-automatic segmentation methods for quantitative image phenotyping in magnetic resonance imaging (MRI) of glioblastoma multiforme (GBM). Methods: MRI examinations with T1 post-gadolinium and FLAIR sequences of 10 GBM patients were downloaded from the Cancer Image Archive site. Two semi-automatic segmentation tools with different algorithms (deformable model and grow cut method) were used to segment contrast enhancement, necrosis and edema regions by two independent observers. A total of 21 imaging features consisting of area and edge groups were extracted automatically from the segmented tumor. The inter-observer variability and coefficient of variation (COV) were calculated to evaluate the reproducibility. Results: Inter-observer correlations and coefficient of variation of imaging features with the deformable model ranged from 0.953 to 0.999 and 2.1% to 9.2%, respectively, and the grow cut method ranged from 0.799 to 0.976 and 3.5% to 26.6%, respectively. Coefficient of variation for especially important features which were previously reported as predictive of patient survival were: 3.4% with deformable model and 7.4% with grow cut method for the proportion of contrast enhanced tumor region; 5.5% with deformable model and 25.7% with grow cut method for the proportion of necrosis; and 2.1% with deformable model and 4.4% with grow cut method for edge sharpness of tumor on CE-T1W1. Conclusion: Comparison of two semi-automated tumor segmentation techniques shows reliable image feature extraction for radiogenomic analysis of GBM patients with multiparametric Brain MRI.

60 APPLIED LIFE SCIENCES↗

TU-AB-BRC-05: Creation of a Monte Carlo TrueBeam Model by Reproducing Varian Phase Space Data

Purpose: To create a Varian TrueBeam 6 MV FFF Monte Carlo model using BEAMnrc/EGSnrc that accurately reproduces the Varian representative dataset, followed by tuning the model’s source parameters to accurately reproduce in-house measurements. Methods: A BEAMnrc TrueBeam model for 6 MV FFF has been created by modifying a validated 6 MV Varian CL21EX model. Geometric dimensions and materials were adjusted in a trial and error approach to match the fluence and spectra of TrueBeam phase spaces output by the Varian VirtuaLinac. Once the model’s phase space matched Varian’s counterpart using the default source parameters, it was validated to match 10 × 10 cm{sup 2} Varian representative data obtained with the IBA CC13. The source parameters were then tuned to match in-house 5 × 5 cm{sup 2} PTW microDiamond measurements. All dose to water simulations included detector models to include the effects of volume averaging and the non-water equivalence of the chamber materials, allowing for more accurate source parameter selection. Results: The Varian phase space spectra and fluence were matched with excellent agreement. The in-house model’s PDD agreement with CC13 TrueBeam representative data was within 0.9% local percent difference beyond the first 3 mm. Profile agreement at 10 cm depth was within 0.9% local percent difference and 1.3 mm distance-to-agreement in the central axis and penumbra regions, respectively. Once the source parameters were tuned, PDD agreement with microDiamond measurements was within 0.9% local percent difference beyond 2 mm. The microDiamond profile agreement at 10 cm depth was within 0.6% local percent difference and 0.4 mm distance-to-agreement in the central axis and penumbra regions, respectively. Conclusion: An accurate in-house Monte Carlo model of the Varian TrueBeam was achieved independently of the Varian phase space solution and was tuned to in-house measurements. KO acknowledges partial support by the CREATE Medical Physics Research Training Network grant of the Natural Sciences and Engineering Research Council (Grant number: 432290).

60 APPLIED LIFE SCIENCES↗

TH-CD-207B-11: Multi-Vendor Phantom Study of CT Lung Density Metrics: Is a Reproducibility of Less Than 1 HU Achievable?

Purpose: To standardize the calibration procedures of CT lung density measurements using low-density reference foams in a phantom, and to demonstrate a reproducibility of less than 1 HU for lung equivalent foam densities measured across CT vendor platforms and protocols. Methods: A phantom study was conducted on CT scanner models from 4 vendors at 100, 120, and 135/140 kVp and 1.5, 3, and 6 mGy dose settings, using a lung density phantom containing air, water, and 3 reference foams (indirectly calibrated) with discrete densities simulating a 5-cm slice of the human chest. Customized segmentation software was used to analyze the images and generate a mean HU and variance for each of the density for the 22 vendor/protocols. A 3-step calibration process was devised to remove a scanner-dependent parameter using linear regression of the HU value vs the relative electron density. The results were mapped to a single energy (80 keV) for final comparison. Results: The heterogeneity across vendor platforms for each density assessed by a random effects model was reduced by 50% after re-calibration, while the standard deviation of the mean HU values also improved by about the same amount. The 95% CI of the final HU value was within +/−1 HU for all 3 reference foam densities. For the backing lung foam in the phantom (served as an “unknown”), this CI is +/− 1.6 HU. The kVp and dose settings did not appear to have significant contributions to the variability. Conclusion: With the proposed calibration procedures, the inter-scanner reproducibility of better than 1 HU is demonstrated in the current phantom study for the reference foam densities, but not yet achieved for a test density. The sources of error are being investigated in the next round of scanning with a certified Standard Reference Material for direct calibration. Fain: research funding from GE Healthcare to develop pulmonary MRI techniques. Hoppel: employee of Toshiba Medical Research Institute USA/financial interest with GE Healthcare. M. Fuld: employee of Siemens Healthcare for medical device equipment and software. This project is supported partially by RSNA QIBA Concept Award (Fain), NIH/NIBIB, HHSN268201300071C (Y).

60 APPLIED LIFE SCIENCES↗

MOLNs: A Cloud Platform for Interactive, Reproducible, and Scalable Spatial Stochastic Computational Experiments in Systems Biology Using PyURDME

Computational experiments using spatial stochastic simulations have led to important new biological insights, but they require specialized tools and a complex software stack, as well as large and scalable compute and data analysis resources due to the large computational cost associated with Monte Carlo computational workflows. The complexity of setting up and managing a large-scale distributed computation environment to support productive and reproducible modeling can be prohibitive for practitioners in systems biology. This results in a barrier to the adoption of spatial stochastic simulation tools, effectively limiting the type of biological questions addressed by quantitative modeling. In this paper, we present PyURDME, a new, user-friendly spatial modeling and simulation package, and MOLNs, a cloud computing appliance for distributed simulation of stochastic reaction-diffusion models. In conclusion, MOLNs is based on IPython and provides an interactive programming platform for development of sharable and reproducible distributed parallel computational experiments.

97 MATHEMATICS AND COMPUTING↗

A multivariate approach to ensure statistical reproducibility of climate model simulations

Effective utilization of novel hybrid architectures of pre-exascale and exascale machines requires transformations to global climate modeling systems that may not reproduce the original model solution bit-for-bit. Round-off level differences grow rapidly in these non-linear and chaotic systems. This makes it difficult to isolate bugs/errors from innocuous growth expected from round-off level differences. Here, we apply two modern multivariate two sample equality of distribution tests to evaluate statistical reproducibility of global climate model simulations using standard monthly output of short (~ 1-year) simulation ensembles of a control model and a modified model of US Department of Energy's Energy Exascale Earth System Model (E3SM). Both the tests are able to identify changes induced by modifications to some model tuning parameters. We also conduct formal power analyses of the tests by applying them on designed suites of short simulation ensembles each with an increasingly different climate from the control ensemble. The results are compared against those from another such test. These power analyses provide a framework to quantify the degree of differences that can be detected confidently by the tests for a given ensemble size (sample size). This will allow model developers using the tests to make an informed decision when accepting/rejecting an unintentional non-bit-for-bit change to the model solution.

Mahajan, Salil↗

Reproduced Computational Results Report for “Ginkgo: A Modern Linear Operator Algebra Framework for High Performance Computing”

The article titled “Ginkgo: A Modern Linear Operator Algebra Framework for High Performance Computing” by Anzt et al. presents a modern, linear operator centric, C++ library for sparse linear algebra. Experimental results in the article demonstrate that Ginkgo is a flexible and user-friendly framework capable of achieving high-performance on state-of-the-art GPU architectures. In this report, the Ginkgo library is installed and a subset of the experimental results are reproduced. Specifically, the experiment that shows the achieved memory bandwidth of the Ginkgo Krylov linear solvers on NVIDIA A100 and AMD MI100 GPUs is redone and the results are compared to what presented in the published article. Upon completion of the comparison, the published results are deemed reproducible.

97 MATHEMATICS AND COMPUTING↗

Experimental Benchmarking for High-Reproducibility, Cross-Institutional Evaluation of Iron Redox Electrochemistry

We present a practical case study standardizing experimental protocols between collaborators with the goal of understanding ferrous iron (Fe 2+ ) chemistry and improving the iron deposition reaction for energy-efficient, electrochemical iron production. The study of iron reactions can be difficult, as aqueous iron electrolytes exhibit complex behaviors that can lead to differing interpretation of ostensibly similar experiments. The question we want to answer: are we studying the same chemistry? Our protocols address inherent challenges such as the tendency for Fe 2+ to spontaneously oxidize to ferric iron (Fe 3+ ) and the production of hydrogen at the potentials of interest. Our standardized protocol, executed by four collaborators in different labs and institutions, yields high-reproducibility results, and identified glassy carbon electrode surface quality and Fe 3+ impurities in the salt as key factors with outsized effects on cyclic voltammetry measurements. The process of developing the protocols helped to troubleshoot underlying issues that created poorer reversibility and reproducibility. This study highlights the fact that even nominally straightforward electrochemical systems can yield vastly different outcomes due to small differences in experimental preparation and serves as a useful example for creating transparent and achievable standards for the generation of reliable datasets that can be widely used and shared.

Ketter, Benjamin [Argonne National Laboratory (ANL↗