Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data science workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

WormBase in 2022—data, processes, and tools for analyzing Caenorhabditis elegans

WormBase (www.wormbase.org) is the central repository for the genetics and genomics of the nematode Caenorhabditis elegans. We provide the research community with data and tools to facilitate the use of C. elegans and related nematodes as model organisms for studying human health, development, and many aspects of fundamental biology. Throughout our 22-year history, we have continued to evolve to reflect progress and innovation in the science and technologies involved in the study of C. elegans. We strive to incorporate new data types and richer data sets, and to provide integrated displays and services that avail the knowledge generated by the published nematode genetics literature. Here, we provide a broad overview of the current state of WormBase in terms of data type, curation workflows, analysis, and tools, including exciting new advances for analysis of single-cell data, text mining and visualization, and the new community collaboration forum. Concurrently, we continue the integration and harmonization of infrastructure, processes, and tools with the Alliance of Genome Resources, of which WormBase is a founding member.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Identification and correction of temporal and spatial distortions in scanning transmission electron microscopy

Scanning transmission electron microscopy (STEM) has become the technique of choice for quantitative characterization of atomic structure of materials, where the minute displacements of atomic columns from high-symmetry positions can be used to map strain, polarization, octahedra tilts, and other physical and chemical order parameter fields. The latter can be used as inputs into mesoscopic and atomistic models, providing insight into the correlative relationships and generative physics of materials on the atomic level. However, these quantitative applications of STEM necessitate understanding the microscope induced image distortions and developing the pathways to compensate them both as part of a rapid calibration procedure for in situ imaging, and the post-experimental data analysis stage. Here, we explore the spatiotemporal structure of the microscopic distortions in STEM using multivariate analysis of the atomic trajectories in the image stacks. Based on the behavior of principal component analysis (PCA), we develop the Gaussian process (GP)-based regression method for quantification of the distortion function. The limitations of such an approach and possible strategies for implementation as a part of in-line data acquisition in STEM are discussed. Here, the analysis workflow is summarized in a Jupyter notebook that can be used to retrace the analysis and analyze the reader's data.

36 MATERIALS SCIENCE↗
DOE OSTI2024

3D Geologic Framework Modelling of the Los Alamos National Laboratory Site and Pajarito Plateau: Integrating a realistic 3D fault network and modelling subsurface relationships in a sparsely sampled and complex geologic region

The subsurface geology beneath the Pajarito Plateau is critical to understanding the seismic hazard of the Pajarito Fault System, yet our understanding of this geology is relatively poor. While previous 3D geologic framework models of the area have been created for the purposes of understanding hydrogeologic flow, they are inadequate for the purposes of understanding the Pajarito Fault System. The specific challenges of using oil and gas software for this purpose include: (1) the geologic complexities resulting from volcanism and tectonism; (2) a need for a high level of stratigraphic detail over a large area; (3) a near complete lack of seismic data; and (4) sparse wellbore data. Presented here is a workflow that handles these challenges of adapting commercially available software used by the oil and gas industries to this seismic hazard problem.

58 GEOSCIENCES↗
DOE OSTI2020

Viral Annotation Pipeline in KBase

This data analysis workflow demonstrates how to process a metagenome to identify viruses using VirSorter. Then, provides taxonomic classification using vConTACT2.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Initial User Study Insights Clarified Community Requirements & Early Feedback on Conceptual Design

The HPDF User Experience (UX) team conducted eight semi-structured interviews with twelve individuals leading data and computing infrastructure work at DOE Office of Science (SC) user facilities or projects. This report presents key takeaways synthesizing community perspectives and needs, along with recommendations for the project. These insights should be considered as conceptual design work continues, early partnerships begin, workflow readiness activities evaluate and enhance key scientific tools, and early access systems are made available to the Office of Science (SC) community. Vigilance in meeting community requirements and ensuring workflow readiness will be needed to ensure HPDF’s success.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

DYFLOW: A flexible framework for orchestrating scientific workflows on supercomputers

Modern scientific workflows are increasing in complexity with growth in computation power, incorporation of non-traditional computation methods, and advances in technologies enabling data streaming to support on-the-fly computation. These workflows have unpredictable runtime behaviors, and a fixed, predetermined resource assignment on supercomputers can be inefficient for overall performance and throughput. Inability to change resource assignments further limits the scientists to avail of science-driven opportunities or respond to failures.We introduce DYFLOW, a flexible framework that orchestrates scientific workflows on supercomputers based on user-designed policies. DYFLOW compartmentalizes orchestration stages into simplified constructs, and end-users can program and reuse them according to their workflow requirements through an easy-to-use interface. These constructs hide the intricacies involved in runtime management from end-users, for instance, procurement of information to understand the workflow state, assessment, and supervision of the runtime changes. DYFLOW is designed to work alongside existing workflow management systems and reuse the available (static) support for workflow management. We have integrated DYFLOW with an existing workflow management tool as a demonstration. With experiments performed on use cases from three types of scientific workflows and two different parallel architectures, we show that DYFLOW achieves the desired orchestration incurring a small cost to carry out the runtime changes.

Singhal, Swati↗
DOE OSTI2020

AI-Science for Performance Optimization and Diagnosis of Science Instrument Federations

Next generation of science workflows are expected to be executed over complex federations composed of supercomputers, science instruments, storage systems and networks, with new additions of the edge and cloud systems and services. The sheer complexity of these multi-domain federations makes it hard to manage them and optimize their performance, as small impedance mismatches (that can dynamically develop between systems) could drastically degrade the entire federation performance. Recent proliferation of Software Defined Everything (SDX) technologies combined with containerization frameworks provide custom instruments that can monitor and collect critical measurements at various levels to support diagnoses and performance optimization; but their data too enormous for human operators and analysts to process and generate decisions. Machine Learning (ML) methods that extract critical parameters, relationships and trends from the data offer general solutions. Artificial Intelligence (AI) and ML methods must be custom-developed for these problems based on solid, rigorous foundations, since black-box approaches are often ineffective and unsound.We propose to develop comprehensive AI-Science for the performance of science federations to (i) monitor and control storage, networks, experiments, and computing systems across multiple domains via softwarization layers, at speeds and scales orders of magnitude superior to current practice, (ii) optimally realize and orchestrate complex workflows with high performance by using dynamic state and performance estimation methods, and (iii) aggregate measurements across sites and time to develop infrastructure-level profiles, optimizations and diagnoses using AI-Science based on foundational principles from ML, game theory, and information fusion areas.

Rao, Nageswara↗
DOE OSTI2024

Validation of a synthetic fast ion loss detector model for Wendelstein 7-X

Abstract We present the first validated synthetic diagnostic for fast ion loss detectors (FILDs) in the Wendelstein 7-X (W7-X) stellarator. This model has been developed on, and validated against experimental data from, a FILD provided by the National Institute for Fusion Science (NIFS-FILD), with potential future applicability to the existing Faraday Cup FILD (FC-FILD) on W7-X as well as the scintillating FILD (S-FILD) currently under development. A workflow combining Monte Carlo codes BEAMS3D and ASCOT5 is used to track fast ions produced by neutral beam injection from the moment of ionization until they are thermalized or lost from the last closed flux surface, and from there to a virtual plane which serves as a projection of the entrance aperture to the FILD. Simulations in ASCOT5 are analyzed via a geometric method to determine the probability of transmission through the FILD aperture and onto the detector as a function of normalized momentum, pitch angle, gyrophase, and position at the virtual plane. This probability is then applied to the simulated ions arriving from the plasma, producing a simulated signal from a computationally tractable number of simulated fast ions. Simulated signals are presented for two W7-X experiments with neutral beam injection and quantitatively compared with experimental measurements from the NIFS-FILD diagnostic. An estimate of the frequency of charge-exchange with neutral particles in the edge is performed, and it is found that this process may have a significant impact on the measured signals.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

Enabling Command-and-Control in Advanced In Situ Workflows

Scientific discovery is progressing towards autonomous science with the combination of scientific instruments, high-performance computing, and artificial intelligence in complex workflows. This evolution introduces new requirements for managing scientific workflows, including feedback loops, near real-time constraints, and the ability to dynamically control workflow execution. In situ workflows that analyze and visualize data as it is generated are well-suited to satisfy stringent time constraints and their iterative nature offers greater opportunities for command-and-control. However, only a few of the many workflow management systems available have been specifically designed to manage in situ workflows and often lack support for automated feedback loops that allow analysis and visualization components to interact with the main scientific data producer. To address this need, we present in this paper how to add command-and-control capabilities to a workflow management system. We identify the functional design requirements of such a command-and-control system, detail its architecture, interface, and core mechanisms, and illustrate how advanced in situ workflows can leverage command-and-control in three use cases: graceful termination with checkpoint, dynamic and adaptive data reduction, and event-triggered analysis.

Mehta, Kshitij [ORNL] (ORCID:0000000297149981)↗
DOE OSTI2023

Toward implementing autonomous adaptive data acquisition for scanning hyperspectral imaging of biological systems

Autonomous experimentation is an emerging area of research, primarily related to autonomous vehicles, scientific combinatorial discovery approaches in materials science and drug discovery, and iterative research loops of planning, experimentation, and analysis. However, autonomous approaches developed in these contexts are difficult to apply to high-dimensional mapping technologies, such as scanning hyperspectral imaging of biological systems, due to sample complexity and heterogeneity. We briefly cover the history of adaptive sampling algorithms and surrogate modeling in order to define autonomous adaptive data acquisition as an objective-based, flexible building block for future biological imaging experimentation driven by intelligent infrastructure. We subsequently summarize the recent implementations of autonomous adaptive data acquisition (AADA) for scanning hyperspectral imaging, assess how these address the difficulties of autonomous approaches in hyperspectral imaging, and highlight the AADA design variation from a goal-oriented perspective. Finally, we present a modular AADA architecture that embeds AADA-driven flexible building blocks to address the challenge of time resolution for high-dimensional scanning hyperspectral imaging of nonequilibrium dynamical systems. In our example research-driven experimental design case, we propose an AADA infrastructure for time-resolved, noninvasive, and label-free scanning hyperspectral imaging of living biological systems. This AADA infrastructure can accurately target the correct state of the system for experimental workflows that utilize subsequent expensive, high-information-content analytical techniques.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

HTESP (High-throughput electronic structure package): A package for high-throughput ab initio calculations

High-throughput ab initio calculations are the indispensable parts of data-driven discovery of new materials with desirable properties, as reflected in the establishment of several online material databases. The accumulation of extensive theoretical data through computations enables data-driven discovery by constructing machine learning and artificial intelligence models to predict novel compounds and forecast their properties. Efficient usage and extraction of data from these existing online material databases can accelerate the next stage materials discovery that targets different and more advanced properties, such as electron–phonon coupling for phonon-mediated superconductivity. However, extracting data from these databases, generating tailored input files for different ab initio calculations, performing such calculations, and analyzing new results can be demanding tasks. Here, in this work, we introduce a software package named “HTESP” (High-Throughput Electronic Structure Package) written in Python and Bash languages, which automates the entire workflow including data extraction, input file generation, calculation submission, result collection and plotting. Our HTESP will help speed up future computational materials discovery processes.

36 MATERIALS SCIENCE↗
DOE OSTI2022

rabpro: global watershed boundaries, river elevation profiles, and catchment statistics

River and Basin Profiler (rabpro) is a Python package to delineate watersheds, extract river flowlines and elevation profiles, and compute watershed statistics for any location on the Earth’s surface. As fundamental hydrologically-relevant units of surface area, watersheds are areas of land that drain via aboveground pathways to the same location, or outlet. Delineations of watershed boundaries are typically performed on digital elevation models (DEMs) that represent surface elevations as gridded rasters. Depending on the resolution of the DEM and the size of the watershed, delineation may be very computationally expensive. With this in mind, we designed rabpro to provide user-friendly workflows to manage the complexity and computational expense of watershed calculations given an arbitrary coordinate pair. In addition to basic watershed delineation, rabpro will extract the elevation profile for a watershed’s mainchannel flowline. This enables the computation of river slope, which is a critical parameter in many hydrologic and geomorphologic models. Finally, rabpro provides a user-friendly wrapper around Google Earth Engine’s (GEE) Python API to enable cloud-computing of zonal watershed statistics and/or time-varying forcing data from hundreds of available datasets. Altogether, rabpro provides the ability to automate or semi-automate complex watershed analysis workflows across broad spatial extents.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Evaluation & Development of Algorithms & Techniques for Streaming Detector Readout

The advancement in microelectronics capabilities, computing, and data science in the last decade has been remarkable. With the start of the 12 GeV Science Program at Jefferson Lab (JLab) and the conceptual detector design for the upcoming Ion Collider (EIC) in mind, we have evaluated and developed the possibility of evolving and improving the existing nuclear science research workflow based on these advances. Specifically, we have prototyped components of streaming readout.

43 PARTICLE ACCELERATORS↗
DOE OSTI2022

Metabolome patterns identify active dechlorination in bioaugmentation consortium SDC-9™

Ultra-high performance liquid chromatography–high-resolution mass spectrometry (UPHLC–HRMS) is used to discover and monitor single or sets of biomarkers informing about metabolic processes of interest. The technique can detect 1000’s of molecules (i.e., metabolites) in a single instrument run and provide a measurement of the global metabolome, which could be a fingerprint of activity. Despite the power of this approach, technical challenges have hindered the effective use of metabolomics to interrogate microbial communities implicated in the removal of priority contaminants. Herein, our efforts to circumvent these challenges and apply this emerging systems biology technique to microbiomes relevant for contaminant biodegradation will be discussed. Chlorinated ethenes impact many contaminated sites, and detoxification can be achieved by organohalide-respiring bacteria, a process currently assessed by quantitative gene-centric tools (e.g., quantitative PCR). This laboratory study monitored the metabolome of the SDC-9™ bioaugmentation consortium during cis-1,2-dichloroethene (cDCE) conversion to vinyl chloride (VC) and nontoxic ethene. Untargeted metabolomics using an UHPLC-Orbitrap mass spectrometer and performed on SDC-9™ cultures at different stages of the reductive dechlorination process detected ~10,000 spectral features per sample arising from water-soluble molecules with both known and unknown structures. Multivariate statistical techniques including partial least squares-discriminate analysis (PLSDA) identified patterns of measurable spectral features (peak patterns) that correlated with dechlorination (in)activity, and ANOVA analyses identified 18 potential biomarkers for this process. Statistical clustering of samples with these 18 features identified dechlorination activity more reliably than clustering of samples based only on chlorinated ethene concentration and Dhc 16S rRNA gene abundance data, highlighting the potential value of metabolomic workflows as an innovative site assessment and bioremediation monitoring tool.

environmental monitoring↗
DOE OSTI2023

How Climate and Data Quality Impact Photovoltaic Performance Loss Rate Estimations

Different data pipelines and statistical methods are applied to photovoltaic (PV) performance datasets to quantify the performance loss rate (PLR). Since the real values of PLR are unknown, a variety of unvalidated values are reported. As such, the PV industry commonly assumes PLR based on statistically extracted ranges from the literature. However, the accuracy and uncertainty of PLR depend on several parameters including seasonality, local climatic conditions, and the response of a particular PV technology. In addition, the specific data pipeline and statistical method used affect the accuracy and uncertainty. To provide insights, a framework of (≈200 million) synthetic simulations of PV performance datasets using data from different climates is developed. Time series with known PLR and data quality are synthesized, and large parametric studies are conducted to examine the accuracy and uncertainty of different statistical approaches over the contiguous US, with an emphasis on the publicly available and “standardized” library, RdTools . In the results, it is confirmed that PLRs from RdTools are unbiased on average, but the accuracy and uncertainty of individual PLR estimates vary with climate zone, data quality, PV technology, and choice of analysis workflow. Best practices and improvement recommendations based on the findings of this study are provided.

14 SOLAR ENERGY↗
DOE OSTI2020

Volumetric analysis and mesh generation of real and artificial microstructural geometries

Producing a viable finite element mesh of realistic microstructural structural geometry is a critical step in analyzing the thermo-mechanical behavior of complex multi-material composites. Advancements in imaging technology such as micro computed tomography have allowed modelers to access high resolution mesoscale geometries for direct numerical simulation. However, converting from voxel based 3D images to usable finite element meshes has been challenging. A robust method including algorithms and software scripts for generating finite element meshes from 3D imaged microstructures is presented in this paper. It includes a routine for inserting cohesive elements around material interfaces to enable modeling of interface properties including delamination and damage. The algorithms and procedures presented in this method leverage currently available software packages for processing surface based geometry into volume based meshes. In addition to converting real geometry from physical imaging systems, algorithms for producing numerically generated and statistically equivalent microstructural geometry are also included. These artificial microstructures can be a valuable resource for modelers when physical specimens do not exist or are limited in quantity: Method establishes a workflow from voxel data to viable finite element mesh including interface information, and Includes a method for synthetic geometry generation based on metrics from real microstructures.

36 MATERIALS SCIENCE↗
DOE OSTI2026

JGI Archive and Metadata Organizer (JAMO) v2.0.0

JAMO (JGI Archive and Metadata Organizer) helps researchers keep large collections of scientific files organized, findable, and safe. It lets you submit files with consistent, template-driven metadata, bundle related files into sets, and track them as a group instead of one by one. As data ages, JAMO automatically moves it from fast disk to cost-saving tape and can bring it back when needed, keeping storage lean without losing access. A simple web/CLI workflow supports submitting, checking status, retrying, and updating metadata. Compared with generic storage, JAMO's strengths are: clear, searchable metadata tuned for science; set-level organization that mirrors real projects; and built-in lifecycle care (archive, purge, restore) so you don't have to manage those steps yourself.

Cassol, Daniela [Lawrence Berkeley National Labora↗
DOE OSTI2022

Scanning structural mapping at the Life Science X-ray Scattering Beamline

This work describes the instrumentation and software for microbeam scattering and structural mapping at the Life Science X-ray Scattering (LiX) beamline at NSLS-II. Using a two-stage focusing scheme, an adjustable beam size between a few micrometres and a fraction of a millimetre is produced at the sample position. Scattering data at small and wide angles are collected simultaneously on multiple Pilatus detectors. A recent addition of an in-vacuum Pilatus 900k detector, with the detector modules arranged in a C-shaped configuration, has improved the azimuthal angle coverage in the wide-angle data. As an option, fluorescence data can be collected simultaneously. Fly scans have been implemented to minimize the time interval between scattering patterns and to avoid unnecessary radiation damage to the sample. For weakly scattering samples, an in-vacuum sample environment has been developed here to minimize background scattering. Data processing for these measurements is highly sample-specific. To establish a generalized data process workflow, first the data are reduced to reciprocal coordinates at the time of data collection. The users can then quantify features of their choosing from these intermediate data and construct structural maps. As examples, results from in-vacuum mapping of onion epidermal cell walls and 2D tomographic sectioning of an intact poplar stem are presented.

36 MATERIALS SCIENCE↗