Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data science workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

WormBase in 2022—data, processes, and tools for analyzing Caenorhabditis elegans

WormBase (www.wormbase.org) is the central repository for the genetics and genomics of the nematode Caenorhabditis elegans. We provide the research community with data and tools to facilitate the use of C. elegans and related nematodes as model organisms for studying human health, development, and many aspects of fundamental biology. Throughout our 22-year history, we have continued to evolve to reflect progress and innovation in the science and technologies involved in the study of C. elegans. We strive to incorporate new data types and richer data sets, and to provide integrated displays and services that avail the knowledge generated by the published nematode genetics literature. Here, we provide a broad overview of the current state of WormBase in terms of data type, curation workflows, analysis, and tools, including exciting new advances for analysis of single-cell data, text mining and visualization, and the new community collaboration forum. Concurrently, we continue the integration and harmonization of infrastructure, processes, and tools with the Alliance of Genome Resources, of which WormBase is a founding member.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Identification and correction of temporal and spatial distortions in scanning transmission electron microscopy

Scanning transmission electron microscopy (STEM) has become the technique of choice for quantitative characterization of atomic structure of materials, where the minute displacements of atomic columns from high-symmetry positions can be used to map strain, polarization, octahedra tilts, and other physical and chemical order parameter fields. The latter can be used as inputs into mesoscopic and atomistic models, providing insight into the correlative relationships and generative physics of materials on the atomic level. However, these quantitative applications of STEM necessitate understanding the microscope induced image distortions and developing the pathways to compensate them both as part of a rapid calibration procedure for in situ imaging, and the post-experimental data analysis stage. Here, we explore the spatiotemporal structure of the microscopic distortions in STEM using multivariate analysis of the atomic trajectories in the image stacks. Based on the behavior of principal component analysis (PCA), we develop the Gaussian process (GP)-based regression method for quantification of the distortion function. The limitations of such an approach and possible strategies for implementation as a part of in-line data acquisition in STEM are discussed. Here, the analysis workflow is summarized in a Jupyter notebook that can be used to retrace the analysis and analyze the reader's data.

36 MATERIALS SCIENCE↗
DOE OSTI2024

3D Geologic Framework Modelling of the Los Alamos National Laboratory Site and Pajarito Plateau: Integrating a realistic 3D fault network and modelling subsurface relationships in a sparsely sampled and complex geologic region

The subsurface geology beneath the Pajarito Plateau is critical to understanding the seismic hazard of the Pajarito Fault System, yet our understanding of this geology is relatively poor. While previous 3D geologic framework models of the area have been created for the purposes of understanding hydrogeologic flow, they are inadequate for the purposes of understanding the Pajarito Fault System. The specific challenges of using oil and gas software for this purpose include: (1) the geologic complexities resulting from volcanism and tectonism; (2) a need for a high level of stratigraphic detail over a large area; (3) a near complete lack of seismic data; and (4) sparse wellbore data. Presented here is a workflow that handles these challenges of adapting commercially available software used by the oil and gas industries to this seismic hazard problem.

58 GEOSCIENCES↗
DOE OSTI2020

Viral Annotation Pipeline in KBase

This data analysis workflow demonstrates how to process a metagenome to identify viruses using VirSorter. Then, provides taxonomic classification using vConTACT2.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Initial User Study Insights Clarified Community Requirements & Early Feedback on Conceptual Design

The HPDF User Experience (UX) team conducted eight semi-structured interviews with twelve individuals leading data and computing infrastructure work at DOE Office of Science (SC) user facilities or projects. This report presents key takeaways synthesizing community perspectives and needs, along with recommendations for the project. These insights should be considered as conceptual design work continues, early partnerships begin, workflow readiness activities evaluate and enhance key scientific tools, and early access systems are made available to the Office of Science (SC) community. Vigilance in meeting community requirements and ensuring workflow readiness will be needed to ensure HPDF’s success.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

DYFLOW: A flexible framework for orchestrating scientific workflows on supercomputers

Modern scientific workflows are increasing in complexity with growth in computation power, incorporation of non-traditional computation methods, and advances in technologies enabling data streaming to support on-the-fly computation. These workflows have unpredictable runtime behaviors, and a fixed, predetermined resource assignment on supercomputers can be inefficient for overall performance and throughput. Inability to change resource assignments further limits the scientists to avail of science-driven opportunities or respond to failures.We introduce DYFLOW, a flexible framework that orchestrates scientific workflows on supercomputers based on user-designed policies. DYFLOW compartmentalizes orchestration stages into simplified constructs, and end-users can program and reuse them according to their workflow requirements through an easy-to-use interface. These constructs hide the intricacies involved in runtime management from end-users, for instance, procurement of information to understand the workflow state, assessment, and supervision of the runtime changes. DYFLOW is designed to work alongside existing workflow management systems and reuse the available (static) support for workflow management. We have integrated DYFLOW with an existing workflow management tool as a demonstration. With experiments performed on use cases from three types of scientific workflows and two different parallel architectures, we show that DYFLOW achieves the desired orchestration incurring a small cost to carry out the runtime changes.

Singhal, Swati↗
DOE OSTI2020

AI-Science for Performance Optimization and Diagnosis of Science Instrument Federations

Next generation of science workflows are expected to be executed over complex federations composed of supercomputers, science instruments, storage systems and networks, with new additions of the edge and cloud systems and services. The sheer complexity of these multi-domain federations makes it hard to manage them and optimize their performance, as small impedance mismatches (that can dynamically develop between systems) could drastically degrade the entire federation performance. Recent proliferation of Software Defined Everything (SDX) technologies combined with containerization frameworks provide custom instruments that can monitor and collect critical measurements at various levels to support diagnoses and performance optimization; but their data too enormous for human operators and analysts to process and generate decisions. Machine Learning (ML) methods that extract critical parameters, relationships and trends from the data offer general solutions. Artificial Intelligence (AI) and ML methods must be custom-developed for these problems based on solid, rigorous foundations, since black-box approaches are often ineffective and unsound.We propose to develop comprehensive AI-Science for the performance of science federations to (i) monitor and control storage, networks, experiments, and computing systems across multiple domains via softwarization layers, at speeds and scales orders of magnitude superior to current practice, (ii) optimally realize and orchestrate complex workflows with high performance by using dynamic state and performance estimation methods, and (iii) aggregate measurements across sites and time to develop infrastructure-level profiles, optimizations and diagnoses using AI-Science based on foundational principles from ML, game theory, and information fusion areas.

Rao, Nageswara↗
NASA NTRS2015

Technical Challenges and Opportunities of Centralizing Space Science Mission Operations (SSMO) at NASA Goddard Space Flight Center

The NASA Goddard Space Science Mission Operations project (SSMO) is performing a technical cost-benefit analysis for centralizing and consolidating operations of a diverse set of missions into a unified and integrated technical infrastructure. The presentation will focus on the notion of normalizing spacecraft operations processes, workflows, and tools. It will also show the processes of creating a standardized open architecture, creating common security models and implementations, interfaces, services, automations, notifications, alerts, logging, publish, subscribe and middleware capabilities. The presentation will also discuss how to leverage traditional capabilities, along with virtualization, cloud computing services, control groups and containers, and possibly Big Data concepts.

Science↗
DOE OSTI2024

Validation of a synthetic fast ion loss detector model for Wendelstein 7-X

Abstract We present the first validated synthetic diagnostic for fast ion loss detectors (FILDs) in the Wendelstein 7-X (W7-X) stellarator. This model has been developed on, and validated against experimental data from, a FILD provided by the National Institute for Fusion Science (NIFS-FILD), with potential future applicability to the existing Faraday Cup FILD (FC-FILD) on W7-X as well as the scintillating FILD (S-FILD) currently under development. A workflow combining Monte Carlo codes BEAMS3D and ASCOT5 is used to track fast ions produced by neutral beam injection from the moment of ionization until they are thermalized or lost from the last closed flux surface, and from there to a virtual plane which serves as a projection of the entrance aperture to the FILD. Simulations in ASCOT5 are analyzed via a geometric method to determine the probability of transmission through the FILD aperture and onto the detector as a function of normalized momentum, pitch angle, gyrophase, and position at the virtual plane. This probability is then applied to the simulated ions arriving from the plasma, producing a simulated signal from a computationally tractable number of simulated fast ions. Simulated signals are presented for two W7-X experiments with neutral beam injection and quantitatively compared with experimental measurements from the NIFS-FILD diagnostic. An estimate of the frequency of charge-exchange with neutral particles in the edge is performed, and it is found that this process may have a significant impact on the measured signals.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
NASA NTRS2024

Optimizing Sample Collection and Accessibility through the Biospecimen and Tissue Sharing Collection (BTSC) Program

The Space Radiation Element (SRE) of the Human Research Program (HRP) is dedicated to establishing a robust biospecimen and tissue sharing collection (BTSC) program that enhances sample collection, tracking, access, distribution, and usability, with the goal of maximizing scientific return. By leveraging biospecimens and tissues from previous experiments, HRP effectively achieves its scientific objectives in characterizing and mitigating the human health impacts of spaceflight while optimizing resource utilization. To further improve the usability and accessibility of the current biospecimen archive, the project aims to expand upon NASA's existing resources and institutional knowledge, ensuring ongoing modernization. To facilitate seamless navigation of the program's workflow, an educational series on the BTSC program is provided to Principal Investigators (PIs). This comprehensive series equips PIs with crucial information on submitting their inventory via the BTSC Metadata Intake Form, ultimately leading to the public availability of their data on NASA's Life Science Portal (NLSP). Covering various aspects such as metadata submission instructions and backend processes for transferring metadata to the Laboratory Information Management System (LIMS), the series incorporates guidance from NASA's Biological Institutional Scientific Collection (NBISC) and Ames Life Sciences Data Archive (ALSDA). The BTSC program represents a significant stride towards enhancing the usability and accessibility of biospecimens for space research. By enabling NASA to deepen its understanding of the health implications of long-term spaceflight, this initiative plays a pivotal role in ensuring the safety and well-being of astronauts.

Shelita Renee Augustus↗
DOE OSTI2025

Enabling Command-and-Control in Advanced In Situ Workflows

Scientific discovery is progressing towards autonomous science with the combination of scientific instruments, high-performance computing, and artificial intelligence in complex workflows. This evolution introduces new requirements for managing scientific workflows, including feedback loops, near real-time constraints, and the ability to dynamically control workflow execution. In situ workflows that analyze and visualize data as it is generated are well-suited to satisfy stringent time constraints and their iterative nature offers greater opportunities for command-and-control. However, only a few of the many workflow management systems available have been specifically designed to manage in situ workflows and often lack support for automated feedback loops that allow analysis and visualization components to interact with the main scientific data producer. To address this need, we present in this paper how to add command-and-control capabilities to a workflow management system. We identify the functional design requirements of such a command-and-control system, detail its architecture, interface, and core mechanisms, and illustrate how advanced in situ workflows can leverage command-and-control in three use cases: graceful termination with checkpoint, dynamic and adaptive data reduction, and event-triggered analysis.

Mehta, Kshitij [ORNL] (ORCID:0000000297149981)↗
NASA NTRS2019

Smart Handoffs: Preserving User Context Between Tools and Services Related to NASA's EOSDIS Data Archive

NASA's Earth Observing System Data and Information System (EOSDIS) is tasked with archiving and distributing Earth Observation data across a range of disciplines, including atmospheric science, oceanography, land processes, natural hazards, solar radiance and even socioeconomic aspects relating to the environment. Given the breadth of disciplines and depth of data that EOSDIS provides, the efficient and intuitive discovery and usage of data by a scientist is of paramount importance. An effective data gathering workflow may involve switching from general use discovery tools to a more bespoke services designed specifically for the scientist's discipline. Providing concrete interoperability between such tools could vastly improve the efficiency of a scientist's workflow.

Analytics↗
NASA NTRSDate not supplied

Co-developing a SAR for Disasters Reference Guide for the International Applications Community

Synthetic Aperture Radar (SAR) data are widely applicable for disaster monitoring, especially in Central and South America where atmospheric effects are a frequent obstacle for disaster monitoring. There is a growing interest in the application of SAR data to diverse disasters, ranging from landslides to volcanic eruptions and floods. With this interest comes a growing need for user-friendly guides that facilitate the integration of SAR into existing workflows. A joint effort between AmeriGEO, university networks in the Americas, NASA SERVIR’s Science Coordination Office (SCO), and the NASA Earth Science Disasters Program, culminated in the “SAR for Disasters” Guide. The guide is modeled after the one-page informational sheets created for SERVIR’s SAR Handbook, with the aim to succinctly communicate the key considerations needed to apply relevant analyses using SAR. Developed with contributors from diverse backgrounds, this guide is designed to be a useful reference for experienced technical users and beginners alike.

SAR↗
DOE OSTI2023

Toward implementing autonomous adaptive data acquisition for scanning hyperspectral imaging of biological systems

Autonomous experimentation is an emerging area of research, primarily related to autonomous vehicles, scientific combinatorial discovery approaches in materials science and drug discovery, and iterative research loops of planning, experimentation, and analysis. However, autonomous approaches developed in these contexts are difficult to apply to high-dimensional mapping technologies, such as scanning hyperspectral imaging of biological systems, due to sample complexity and heterogeneity. We briefly cover the history of adaptive sampling algorithms and surrogate modeling in order to define autonomous adaptive data acquisition as an objective-based, flexible building block for future biological imaging experimentation driven by intelligent infrastructure. We subsequently summarize the recent implementations of autonomous adaptive data acquisition (AADA) for scanning hyperspectral imaging, assess how these address the difficulties of autonomous approaches in hyperspectral imaging, and highlight the AADA design variation from a goal-oriented perspective. Finally, we present a modular AADA architecture that embeds AADA-driven flexible building blocks to address the challenge of time resolution for high-dimensional scanning hyperspectral imaging of nonequilibrium dynamical systems. In our example research-driven experimental design case, we propose an AADA infrastructure for time-resolved, noninvasive, and label-free scanning hyperspectral imaging of living biological systems. This AADA infrastructure can accurately target the correct state of the system for experimental workflows that utilize subsequent expensive, high-information-content analytical techniques.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS2004

Grist : grid-based data mining for astronomy

The Grist project is developing a grid-technology based system as a research environment for astronomy with massive and complex datasets. This knowledge extraction system will consist of a library of distributed grid services controlled by a workflow system, compliant with standards emerging from the grid computing, web services, and virtual observatory communities. This new technology is being used to find high redshift quasars, study peculiar variable objects, search for transients in real time, and fit SDSS QSO spectra to measure black hole masses. Grist services are also a component of the 'hyperatlas' project to serve high-resolution multi-wavelength imagery over the Internet. In support of these science and outreach objectives, the Grist framework will provide the enabling fabric to tie together distributed grid services in the areas of data access, federation, mining, subsetting, source extraction, image mosaicking, statistics, and visualization.

grid computing↗
DOE OSTI2024

HTESP (High-throughput electronic structure package): A package for high-throughput ab initio calculations

High-throughput ab initio calculations are the indispensable parts of data-driven discovery of new materials with desirable properties, as reflected in the establishment of several online material databases. The accumulation of extensive theoretical data through computations enables data-driven discovery by constructing machine learning and artificial intelligence models to predict novel compounds and forecast their properties. Efficient usage and extraction of data from these existing online material databases can accelerate the next stage materials discovery that targets different and more advanced properties, such as electron–phonon coupling for phonon-mediated superconductivity. However, extracting data from these databases, generating tailored input files for different ab initio calculations, performing such calculations, and analyzing new results can be demanding tasks. Here, in this work, we introduce a software package named “HTESP” (High-Throughput Electronic Structure Package) written in Python and Bash languages, which automates the entire workflow including data extraction, input file generation, calculation submission, result collection and plotting. Our HTESP will help speed up future computational materials discovery processes.

36 MATERIALS SCIENCE↗
DOE OSTI2022

rabpro: global watershed boundaries, river elevation profiles, and catchment statistics

River and Basin Profiler (rabpro) is a Python package to delineate watersheds, extract river flowlines and elevation profiles, and compute watershed statistics for any location on the Earth’s surface. As fundamental hydrologically-relevant units of surface area, watersheds are areas of land that drain via aboveground pathways to the same location, or outlet. Delineations of watershed boundaries are typically performed on digital elevation models (DEMs) that represent surface elevations as gridded rasters. Depending on the resolution of the DEM and the size of the watershed, delineation may be very computationally expensive. With this in mind, we designed rabpro to provide user-friendly workflows to manage the complexity and computational expense of watershed calculations given an arbitrary coordinate pair. In addition to basic watershed delineation, rabpro will extract the elevation profile for a watershed’s mainchannel flowline. This enables the computation of river slope, which is a critical parameter in many hydrologic and geomorphologic models. Finally, rabpro provides a user-friendly wrapper around Google Earth Engine’s (GEE) Python API to enable cloud-computing of zonal watershed statistics and/or time-varying forcing data from hundreds of available datasets. Altogether, rabpro provides the ability to automate or semi-automate complex watershed analysis workflows across broad spatial extents.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Evaluation & Development of Algorithms & Techniques for Streaming Detector Readout

The advancement in microelectronics capabilities, computing, and data science in the last decade has been remarkable. With the start of the 12 GeV Science Program at Jefferson Lab (JLab) and the conceptual detector design for the upcoming Ion Collider (EIC) in mind, we have evaluated and developed the possibility of evolving and improving the existing nuclear science research workflow based on these advances. Specifically, we have prototyped components of streaming readout.

43 PARTICLE ACCELERATORS↗