Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “reproducible science”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Breaking the reproducibility barrier with standardized protocols for plant–microbiome research

Inter-laboratory replicability is crucial yet challenging in microbiome research. Leveraging microbiomes to promote soil health and plant growth requires understanding underlying molecular mechanisms using reproducible experimental systems. In a global collaborative effort involving five laboratories, we aimed to help advance reproducibility in microbiome studies by testing our ability to replicate synthetic community assembly experiments. Our study compared fabricated ecosystems constructed using two different synthetic bacterial communities, the model grass Brachypodium distachyon, and sterile EcoFAB 2.0 devices. All participating laboratories observed consistent inoculum-dependent changes in plant phenotype, root exudate composition, and final bacterial community structure, where Paraburkholderia sp. OAS925 could dramatically shift microbiome composition. Comparative genomics and exudate utilization linked the pH-dependent colonization ability of Paraburkholderia, which was further confirmed with motility assays. The study provides detailed protocols, benchmarking datasets, and best practices to help advance replicable science and inform future multi-laboratory reproducibility studies.

Novak, Vlastimil↗
DOE OSTI2026

Using AI to Reproduce Neutrino Cross Section Analysis - Prototyping the Neutrino Discovery Platform

The Neutrino Discovery Platform (NDP) aims to accelerate DUNE-era science by making the neutrino program's existing datasets analyzable through fast, reproducible, and auditable workflows. We report a working version of two of its layers, data curation and agentic orchestration, built and tested end to end on MINERvA open data. The guiding lesson throughout is that a cross section is a measurement, and not just a plotted shape, only if it carries a defensible systematic-uncertainty budget, a trustworthy unfolding, and a reproducible record. Using a single medium-energy playlist pair from the MINERvA open-data release (about $2.05\times10^{17}$ protons on target of data), we first reproduced the shapes of two published charged-current inclusive $\nu_\mu$ measurements through a complete extraction ladder: selection, background subtraction, D'Agostini unfolding, efficiency correction, and flux normalization. These shape-level reproductions ran and tracked the published results, but they lacked the systematic-uncertainty machinery that defines a MINERvA cross section. To supply it, we vendored and built the MINERvA Analysis Toolkit and developed a many-universe systematic-uncertainty tool that produces a portable covariance artifact, a parallel event-loop runner, and a per-run auditability harness. Validated against a published covariance release, the toolchain reproduces the released statistical, flux, and muon-energy-scale terms and shows that they account for roughly 63\% of the total variance, with the remainder unreleased. Using this same infrastructure, we then performed a measurement of our own design, the hadronic recoil-energy distribution of low-energy ($E_\nu<2.5$~GeV) charged-current inclusive events, and found data/simulation shape agreement of $\chi^2/\mathrm{ndf}=1.26$. Together these results show that the platform supports original physics and not only reproductions.

Breaux, Auto [Tulane U. (main)]↗
DOE OSTI2021

Resiliency in numerical algorithm design for extreme scale simulations

Here this work is based on the seminar titled ‘Resiliency in Numerical Algorithm Design for Extreme Scale Simulations’ held March 1–6, 2020, at Schloss Dagstuhl, that was attended by all the authors. Advanced supercomputing is characterized by very high computation speeds at the cost of involving an enormous amount of resources and costs. A typical large-scale computation running for 48 h on a system consuming 20 MW, as predicted for exascale systems, would consume a million kWh, corresponding to about 100k Euro in energy cost for executing 10 23 floating-point operations. It is clearly unacceptable to lose the whole computation if any of the several million parallel processes fails during the execution. Moreover, if a single operation suffers from a bit-flip error, should the whole computation be declared invalid? What about the notion of reproducibility itself: should this core paradigm of science be revised and refined for results that are obtained by large-scale simulation? Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to background storage at frequent intervals will create intolerable overheads in runtime and energy consumption. Forecasts show that the mean time between failures could be lower than the time to recover from such a checkpoint, so that large calculations at scale might not make any progress if robust alternatives are not investigated. More advanced resilience techniques must be devised. The key may lie in exploiting both advanced system features as well as specific application knowledge. Research will face two essential questions: (1) what are the reliability requirements for a particular computation and (2) how do we best design the algorithms and software to meet these requirements? While the analysis of use cases can help understand the particular reliability requirements, the construction of remedies is currently wide open. One avenue would be to refine and improve on system- or application-level checkpointing and rollback strategies in the case an error is detected. Developers might use fault notification interfaces and flexible runtime systems to respond to node failures in an application-dependent fashion. Novel numerical algorithms or more stochastic computational approaches may be required to meet accuracy requirements in the face of undetectable soft errors. These ideas constituted an essential topic of the seminar. The goal of this Dagstuhl Seminar was to bring together a diverse group of scientists with expertise in exascale computing to discuss novel ways to make applications resilient against detected and undetected faults. In particular, participants explored the role that algorithms and applications play in the holistic approach needed to tackle this challenge. This article gathers a broad range of perspectives on the role of algorithms, applications and systems in achieving resilience for extreme scale simulations. The ultimate goal is to spark novel ideas and encourage the development of concrete solutions for achieving such resilience holistically.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

IROS 2023 Workshop Report: Draft Guidelines on Manufacturing Procedures, Test Methods and Reporting for Soft Robotics

Soft roboticists are facing challenges with reproducibility, which prevents researchers from making holistic comparisons to prior work, impedes full understanding of results, and forces the need to “reinvent the wheel,” delaying fundamental advances. Reproducibility of results is key to advancing science as well as achieving technology transfer from research laboratories to industrial applications. Recently, a discussion-based workshop dedicated to the topic, “Developing Standard Testing and Reporting Guidelines for Soft Robotics,” was held at IROS 2023 in Detroit, MI. The purpose of this document is to record a set of recommendations and voluntary draft guidelines for soft roboticists concerning fabrication/manufacturing, test procedures, and reporting, which were collectively developed at the workshop. Together as a community, we hope to improve the reporting standards of soft robotics and drive the field as a whole toward more rigorous research practices.

43 PARTICLE ACCELERATORS↗
DOE OSTI2021

Reproducibility Practice in High-Performance Computing: Community Survey Results

The integrity of science and engineering research is grounded in assumptions of rigor and transparency on the part of those engaging in such research. HPC community effort to strengthen rigor and transparency take the form of reproducibility efforts. In a recent survey of the SC conference community, we collected information about the SC Reproducibility Initiative activities. Here, we present the survey results in this paper. Results show that the reproducibility initiative activities have contributed to higher levels of awareness on the part of SC conference technical program participants, and hint at contributing to greater scientific impact for the published papers of the SC conference series. Stringent point-of-manuscript-submission verification is problematic for reasons we point out, as are inherent difficulties of computational reproducibility in HPC. Future efforts should better decouple the community educational goals from goals that specifically strengthen a research work's potential for long-term impact through reuse 5-10 years down the road.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Explaining and predicting human behavior and social dynamics in simulated virtual worlds: reproducibility, generalizability, and robustness of causal discovery methods

Ground Truth program was designed to evaluate social science modeling approaches using simulation test beds with ground truth intentionally and systematically embedded to understand and model complex Human Domain systems and their dynamics Lazer et al. (Science 369:1060–1062, 2020). Our multidisciplinary team of data scientists, statisticians, experts in Artificial Intelligence (AI) and visual analytics had a unique role on the program to investigate accuracy, reproducibility, generalizability, and robustness of the state-of-the-art (SOTA) causal structure learning approaches applied to fully observed and sampled simulated data across virtual worlds. In addition, we analyzed the feasibility of using machine learning models to predict future social behavior with and without causal knowledge explicitly embedded. In this paper, we first present our causal modeling approach to discover the causal structure of four virtual worlds produced by the simulation teams—Urban Life, Financial Governance, Disaster and Geopolitical Conflict. Our approach adapts the state-of-the-art causal discovery (including ensemble models), machine learning, data analytics, and visualization techniques to allow a human-machine team to reverse-engineer the true causal relations from sampled and fully observed data. We next present our reproducibility analysis of two research methods team’s performance using a range of causal discovery models applied to both sampled and fully observed data, and analyze their effectiveness and limitations. We further investigate the generalizability and robustness to sampling of the SOTA causal discovery approaches on additional simulated datasets with known ground truth. Our results reveal the limitations of existing causal modeling approaches when applied to large-scale, noisy, high-dimensional data with unobserved variables and unknown relationships between them. We show that the SOTA causal models explored in our experiments are not designed to take advantage from vasts amounts of data and have difficulty recovering ground truth when latent confounders are present; they do not generalize well across simulation scenarios and are not robust to sampling; they are vulnerable to data and modeling assumptions, and therefore, the results are hard to reproduce. Finally, when we outline lessons learned and provide recommendations to improve models for causal discovery and prediction of human social behavior from observational data, we highlight the importance of learning data to knowledge representations or transformations to improve causal discovery and describe the benefit of causal feature selection for predictive and prescriptive modeling.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Planetary normal mode computation: Parallel algorithms, performance and reproducibility

This report is an extension of work entitled “Computing planetary interior normal modes with a highly parallel polynomial filtering eigensolver.” by Shi et al., [1] originally presented at the SC18 conference. A highly parallel polynomial filtered eigensolver was developed and exploited to calculate the planetary normal modes. The proposed method is ideally suited for computing interior eigenpairs for large-scale eigenvalue problems as it greatly enhances memory and computational efficiency. In this article, the second-order finite element method is used to further improve the accuracy as only the first-order finite element method was deployed in the previous work. The parallel algorithm, its parallel performance up to 20k processors, and the great computational accuracy are illustrated. The reproducibility of the previous work was successfully performed on the Student Cluster Competition at the SC19 conference by several participant teams using a completely different Mars-model dataset on different clusters. Both weak and strong scaling performances of the reproducibility by the participant teams were impressive and encouraging. The analysis and reflection of their results are demonstrated and future direction is discussed.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Interlaboratory Reproducibility of Contour Method Data in a High Strength Aluminum Alloy

The contour method for residual stress measurement has seen significant development, but an experimental reproducibility study utilizing physical samples has not been published. A double-blind reproducibly study is reported, having scope beginning with EDM cutting and ending with residual stress calculation. A reinforced I-beam sample geometry is identified for its unique residual stress profile when extracted from residual stress bearing quenched aluminum bar (7050-T74). Contour measurements are prescribed on a midplane of symmetry with dimensions 24.0 mm by 50.0 mm. Fourteen identically prepared samples are fabricated from a single long bar with well characterized and uniform residual stress. Five samples throughout the bar are identified for planning measurements to validate sample uniformity and overall suitability of the residual stress field. The planning measurements employ a range of techniques: contour method, neutron diffraction, and hole-drilling. Eight samples are distributed to an international group of participants to execute their standard measurement practice. A double-blind process is followed to provide anonymity. Results are provided by eight participants: six being self-similar and two being quite different, the latter set aside as outliers. An average residual stress field is established from non-outlying results and the spatial distribution of reproducibility standard deviation is determined. The average stress field ranges from -60 to 70 MPa and the reproducibility standard deviation averages 8.1 MPa on the measurement plane. The average reproducibility standard deviation is about 3 × larger for points within 1.0 mm of plane boundaries (17.6 MPa) than for the remaining points (6.1 MPa). Reproducibility standard deviation (among different labs) for contour method residual stress measurement is found to be very similar to repeatability standard deviation (in a single lab) reported in prior work. The reproducibility observed here, for the entire measurement process, is also similar to that found in a prior reproducibility study limited to contour method data analysis.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Proliferation of Faulty Materials Data Analysis in the Literature

As a group of subject matter experts in X-ray photoelectron spectroscopy (XPS), we write this letter to raise awareness of the epidemic of poor and incorrect materials data analysis in the Literature. This is a chronic and growing problem with very undesirable consequences. This issue may contribute to what has been identified as a “reproducibility crisis”1,2 that was the subject of a recent U.S. National Academies study. We are using XPS as one example of what we observe to be a systemic issue.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Open Reproducible Electron Microscopy Data Analysis

Electron microscopy (EM) is a cornerstone technique in the materials and biological sciences capable of imaging structures at nano- to atomic-scale resolution. Advances in technologies mean that one acquires datasets at increasing data rates and sizes. These advancements present enormous opportunities for researchers to understand complex systems. However, processing the resulting large-scale, complex data in a reproducible and shareable way is a real challenge for researchers. The building, managing, and maintaining complex workflows in a reproducible manner requires extensive knowledge in several areas outside the researchers’ core skill sets, such as software engineering, data science, and high-performance computing (HPC). Our work demonstrates an innovative approach to solving these problems, enabling reproducible EM data analysis through container encapsulated pipelines. Using modern container technologies, we encapsulate processing elements and connect them using shared memory. We expose user-friendly, advanced algorithms and tools to allow end users to utilize without expert programming skills. The platform enables reproducible, scalable, shareable pipelines for the analysis and visualization of EM data. Focusing on interoperability, we leverage the DOE and other agencies’ existing investments to provide a powerful software platform for EM data analysis.

Harris, Christopher↗
DOE OSTI2023

nmrrr : A Reproducible Workflow for Binning and Visualizing NMR Spectra From Environmental Samples

Nuclear magnetic resonance (NMR) spectroscopy is a useful tool for detection and identification of molecular structural information, with increasing applications in environmental sciences. NMR instrument outputs are however heterogeneous and require extensive post-processing, creating barriers to their use and application by non-specialists. Here, we report on a new open-source R package, nmrrr, that processes and visualizes spectral data obtained from one-dimensional solution-state and solid-state NMR experiments; the package also performs relevant calculations commonly applied in natural organic matter communities, such as computing the relative abundance of various functional groups. We document the package's installation, dependencies, and functions; and provide a standard workflow for processing NMR data. This package is currently available on CRAN and GitHub, and community contributions are welcome.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Integrating HPC, AI, and Workflows for Scientific Data Analysis: Report from Dagstuhl Seminar 23352

The Dagstuhl Seminar 23352, titled “Integrating HPC, AI, and Workflows for Scientific Data Analysis,” held from August 27 to September 1, 2023, was a significant event focusing on the synergy between High-Performance Computing (HPC), Artificial Intelligence (AI), and scientific workflow technologies. The seminar recognized that modern Big Data analysis in science rests on three pillars: workflow technologies for reproducibility and steering, AI and Machine Learning (ML) for versatile analysis, and HPC for handling large data sets. These elements, while crucial, have traditionally been researched separately, leading to gaps in their integration. The seminar aimed to bridge these gaps, acknowledging the challenges and opportunities at the intersection of these technologies. The event highlighted the complex interplay between HPC, workflows, and ML, noting how ML has increasingly been integrated into scientific workflows, thereby enhancing resource demands and bringing new requirements to HPC architectures, like support for GPUs and iterative computations. The seminar also addressed the challenges in adapting HPC for large-scale ML tasks, including in areas like deep learning, and the need for workflow systems to evolve to leverage ML in data analysis fully. Moreover, the seminar explored how ML could optimize scientific workflow systems and HPC operations, such as through improved scheduling and fault tolerance. A key focus was on identifying prestigious use cases of ML in HPC and understanding their unique, unmet requirements. The stochastic nature of ML and its impact on the reproducibility of data analysis on HPC systems was also a topic of discussion.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Plant science history manuscript GitHub repo

This archive contains codes, notebooks, and other information for reproducing the study that links to this file. The original files are in the plant_sci_hist Github repository which will likely be updated when the code base is adopted for other projects. Thus, this archive is provided so a frozen version used for publishing the plant science history manuscript that references this file.

Botany/trend/methods↗
DOE OSTI2022

Improving the performance of PM6 donor polymer by random ternary copolymerization of BDD and DTBT segments

Developing high-performance wide bandgap polymer donors to match the rapidly growing non-fullerene small molecule acceptors (NF-SMAs) is of great importance to further improve the photovoltaic performances of organic solar cells (OSCs). PM6 and D18 are two leading wide bandgap donor polymers, but their performances are highly sensitive to the molecular weights, which have distinct impact on the morphology and thereby the charge generation and recombination processes. Here, in this work, we reported the random ternary copolymerization of PM6 and D18 segments to develop random terpolymers with finely tuned photoelectronic properties and optimized blend morphology for high-performance OSCs. The resulting random terpolymers (PMD-5, PMD-10, PMD-15 and PMD-20) exhibited finely tuned absorptions, energy levels and blend morphology. The optimal morphology of PMD-15:L8-BO contributed to the fast exciton diffusion and dissociation, efficient charge extraction with suppressed recombination. Thus, the power conversion efficiency was improved from 17.22% of the control PM6 devices and 17.47% of the control D18 devices to 18.12% of PMD-15 based devices, which is the top performance of random terpolymers. Moreover, the PMD-15 exhibited much higher batch-to-batch reproducibility at a broad molecular weight range than that of PM6 and D18, showing promising for practical applications.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Towards FAIR Workflows for Federated Experimental Sciences

A de-centralized, peer-to-peer AI metadata framework is demonstrated which can enable end-to-end metadata & lineage tracking for distributed Machine Learning pipelines spanning edge, High Performance Computing, and cloud environments. With a specific example of end-to-end microscopy algorithm and datasets, the proposed method shows how to enable reproducibility, audit trail, provenance of metadata artifacts. The emerging needs of automation in experimental sciences, ML-centric workflows, and FAIR metadata management across federated compute environments is addressed.

machine learning↗
DOE OSTI2023

Benzo[d]thiazole Based Wide Bandgap Donor Polymers Enable 19.54% Efficiency Organic Solar Cells Along with Desirable Batch-to-Batch Reproducibility and General Applicability

The limited selection pool of high-performance wide bandgap (WBG) polymer donors is a bottleneck problem of the nonfullerene acceptor (NFA) based organic solar cells (OSCs) that impedes the further improvement of their photovoltaic performances. In this work, a series of new WBG polymers, namely PH-BTz, PS-BTz, PF-BTz, and PCl-BTz, are developed by using the bicyclic difluoro-benzo[d]thiazole (BTz) as the acceptor block and benzo[1,2-b:4,5-b']dithiophene (BDT) derivatives as the donor units. By introducing S, F, and Cl atoms to the alkylthienyl sidechains on BDT, the resulting polymers exhibit lowered energy levels and enhanced aggregation properties. The fluorinated PBTz-F not only exhibits a low-lying HOMO level, but also has stronger face-on packing order and results in more uniform fibril-like interpenetrating networks in the related PF-BTz:L8-BO blend. A high-power conversion efficiency (PCE) of 18.57% is achieved. Moreover, PBTz-F also exhibits a good batch-to-batch reproducibility and general applicability. In addition, ternary blend OSCs based on the host PBTz-F:L8-BO blend and PM6 guest donor exhibits a further enhanced PCE of 19.54%, which is among the highest values of OSCs.

36 MATERIALS SCIENCE↗
DOE OSTI2022

A molecular dynamics study of laser-excited gold

The structural evolution of laser-excited systems of gold has previously been measured through ultrafast MeV electron diffraction. However, there has been a long-standing inability of atomistic simulations to provide a consistent picture of the melting process, leading to large discrepancies between the predicted threshold energy density for complete melting, as well as the transition between heterogeneous and homogeneous melting. We make use of two-temperature classical molecular dynamics simulations utilizing three highly successful interatomic potentials and reproduce electron diffraction data presented by Mo et al. [Science 360, 1451–1455 (2018)]. We recreate the experimental electron diffraction data, employing both a constant and temperature-dependent electron–ion equilibration rate. In all cases, we are able to match time-resolved electron diffraction data, and find consistency between atomistic simulations and experiments, only by allowing laser energy to be transported away from the interaction region. This additional energy-loss pathway, which scales strongly with laser fluence, we attribute to hot electrons leaving the target on a timescale commensurate with melting.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Towards Lightweight Data Integration Using Multi-Workflow Provenance and Data Observability

Modern large-scale scientific discovery requires multidisciplinary collaboration across diverse computing facilities, including High Performance Computing (HPC) machines and the Edge-to-Cloud continuum. Integrated data analysis plays a crucial role in scientific discovery, especially in the current AI era, by enabling Responsible AI development, FAIR, Reproducibility, and User Steering. However, the heterogeneous nature of science poses challenges such as dealing with multiple supporting tools, cross-facility environments, and efficient HPC execution. Building on data observability, adapter system design, and provenance, we propose MIDA: an approach for lightweight runtime Multi-workflow Integrated Data Analysis. MIDA defines data observability strategies and adaptability methods for various parallel systems and machine learning tools. With observability, it intercepts the dataflows in the background without requiring instrumentation while integrating domain, provenance, and telemetry data at runtime into a unified database ready for user steering queries. We conduct experiments showing end-to-end multi-workflow analysis integrating data from Dask and MLFlow in a real distributed deep learning use case for materials science that runs on multiple environments with up to 276 GPUs in parallel. We show near-zero overhead running up to 100,000 tasks on 1,680 CPU cores on the Summit supercomputer.

Santos Souza, Renan↗