Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

pyTriBeam

SAND2025-01899O pyTriBeam is a software tool that creates automated processes for a scanning electron microscope including workflows for 3D serial sectioning dataset collection, high-res image montaging, and support for custom script use. This includes integration for 3D chemical mapping (EDS) and crystallographic (EBSD) data collection with select supported detectors. The application allows end users to setup and run customizable data collection workflows without requiring expertise in programming. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Hovey, Chad↗
DOE OSTI2016

SU-F-T-99: Data Visualization From a Treatment Planning Tracking System for Radiation Oncology

Purpose: A treatment planning process tracker database with input forms and a TV-viewable display webpage was developed and implemented in our clinic to collect time data points throughout the process. Tracking plan times is important because it directly affects the patient quality of care. Simply, the longer a patient waits after their initial simulation CT for treatment to begin, the more time the cancer has to progress. The tracker helps to drive workflow through the clinic, while the data collected can be used to understand and manage the process to find and eliminate inefficiencies. Methods: The overall process steps tracked are CT-simulation, mark patient, draw normal contours, draw target volumes, create plan, and review/approve plan. Time stamps for task completion were extracted and used to generate a set of clinic metrics, among which include average time for each step in the process split apart by type of treatment, average time to completion for plans started in a given week, and individual overall completion time per plan. Results: Trends have been tracked for fourteen weeks of clinical data (196 plans). On average, drawing normal contours and target volumes is taking 2–5 times as long as creating the plan itself. This is potentially an issue because it could mean the process is taking too long initially, and it could be forcing the planning step to be done in a short amount of time. We also saw from our graphs that there appears to be no clear trend on the average amount of time per plan week-to-week. Conclusion: A tracker of this type has the potential to provide insight into how time is utilized in our clinic. By equipping our dosimetrists, radiation oncologists, and physicists with individualized metric sets, the tracker can help provide visibility and drive workflow. Funded in part by CPRIT (RP140105).

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

LANL Activities on Mechanistic Approach to Analyzing and Improving Unconventional Hydrocarbon Production

Hydrocarbon production from shale reservoirs is inherently inefficient and challenging since these are low permeability plays. In addition, there is a limited understanding of the fundamentals and the controlling mechanisms, further complicating how to optimize these plays. Herein, we summarize our experimental and computational efforts fully and partially supported by the fundamental shale portfolio to reveal unconventional shale fundamentals and devise development strategies to enhance extraction efficiency with a minimal environmental footprint. Integrating these fundamentals with machine learning, we outline a pathway to improve the predictive power of our models, which enhances the forecast quality of production, thereby improving the economics of operations in unconventional reservoirs. For instance, we have developed science informed workflows and platforms for optimizing pressure-drawdown at a site, which allow operators to make reservoir-management decisions that optimize recovery in consideration of future production. Recently, our work relies on the hybridization of physics-based prediction and machine learning, whereby accurate synthetic data (combined with available site data) can enable the application of machine learning methods for rapid forecasting and optimization. Consequently, the workflow and platform are readily extendable to operations at other sites, plays, and basins.

04 OIL SHALES AND TAR SANDS↗
DOE OSTI2017

A graph-based computational framework for simulation and optimisation of coupled infrastructure networks

Here, we present a computational framework that facilitates the construction, instantiation, and analysis of large-scale optimization and simulation applications of coupled energy networks. The framework integrates the optimization modeling package PLASMO and the simulation package DMNetwork (built around PETSc). These tools use a common graphbased abstraction that enables us to achieve compatibility between data structures and to build applications that use network models of different physical fidelity. We also describe how to embed these tools within complex computational workflows using SWIFT, which is a tool that facilitates parallel execution of multiple simulation runs and management of input and output data. We discuss how to use these capabilities to target coupled natural gas and electricity systems.

42 ENGINEERING↗
DOE OSTI2019

TAZeR: Hiding the Cost of Remote I/O in Distributed Scientific Workflows

A perennial bottleneck in distributed workflow analytics is long access latencies for remote data. We ask the question: assuming that data must be accessed remotely, can latencies be hidden? We present TAZeR, a framework that reduces data access latency while increasing data reuse. TAZeR transparently converts POSIX I/O into operations that interleave application work with data transfer, i.e. read, prefetching, and write stage-out. TAZeR ensures read data moves directly to application memory without synchronous intervention (soft zero-copy). TAZeR uses distributed bandwidth-aware staging to exploit reuse across application tasks and to manage the capacity constraints of fast hierarchical storage. We evaluate TAZeR on a High Energy Physics workflow that requests remote data at 48 Gb/s (over two 1 Gb/s WAN links) using complex access patterns. TAZeR is 12× and 22× faster than XRootD (state-of-the-art) and file copies (current approach), respectively; and within 7% of optimal. We discuss conditions when TAZeR can hide I/O accesses; and evaluate performance as effective staging sizes change.

Tazer, remote I/O, data staging↗
DOE OSTI2015

Progress on the FabrIc for Frontier Experiments project at Fermilab

The FabrIc for Frontier Experiments (FIFE) project is an ambitious, major-impact initiative within the Fermilab Scientific Computing Division designed to lead the computing model for Fermilab experiments. FIFE is a collaborative effort between experimenters and computing professionals to design and develop integrated computing models for experiments of varying needs and infrastructure. The major focus of the FIFE project is the development, deployment, and integration of Open Science Grid solutions for high throughput computing, data management, database access and collaboration within experiment. To accomplish this goal, FIFE has developed workflows that utilize Open Science Grid sites along with dedicated and commercial cloud resources. The FIFE project has made significant progress integrating into experiment computing operations several services including new job submission services, software and reference data distribution through CVMFS repositories, flexible data transfer client, and access to opportunistic resources on the Open Science Grid. Hence, the progress with current experiments and plans for expansion with additional projects will be discussed. FIFE has taken a leading role in the definition of the computing model for Fermilab experiments, aided in the design of computing for experiments beyond Fermilab, and will continue to define the future direction of high throughput computing for future physics experiments worldwide

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

The Practical Obstacles of Data Transfer: Why researchers still love scp

The importance of computing facilities is heralded every six months with the announcement of the new Top500 list, showcasing the world s fastest supercomputers. Unfortu- nately, with great computing capability does not come great long-term data storage capacity, which often means users must move their data to their local site archive, to remote sites where they may be doing future computation or anal- ysis, or back to their home institution, else face the dreaded data purge that most HPC centers employ to keep utiliza- tion of large parallel filesystems low to manage performance and capacity. At HPC centers, data transfer is crucial to the scientific workflow and will increase in importance as computing systems grow in size. The Energy Sciences Net- work (ESnet) recently launched its fifth generation network, a 100 Gbps high-performance, unclassified national network connecting more than 40 DOE research sites to support scientific research and collaboration. Despite the tenfold increase in bandwidth to DOE research sites amenable to multiple data transfer streams and high throughput, in prac- tice, researchers often under-utilize the network and resort to painfully-slow single stream transfer methods such as scp to avoid the complexity of using multiple stream tools such as GridFTP and bbcp, and contend with frustration from the lack of consistency of available tools between sites. In this study we survey and assess the data transfer methods pro- vided at several DOE supported computing facilities, includ- ing both leadership-computing facilities, connected through ESnet. We present observed transfer rates, suggested opti- mizations, and discuss the obstacles the tools must overcome to receive wide-spread adoption over scp.

Nam, Hai Ah↗
NASA NTRS2000

Web Time-Management Tool

Oak Grove Reactor, developed by Oak Grove Systems, is a new software program that allows users to integrate workflow processes. It can be used with portable communication devices. The software can join e-mail, calendar/scheduling and legacy applications into one interactive system via the web. Priority tasks and due dates are organized and highlighted to keep the user up to date with developments. Reactor works with existing software and few new skills are needed to use it. Using a web browser, a user can can work on something while other users can work on the same procedure or view its status while it is being worked on at another site. The software was developed by the Jet Propulsion Lab and originally put to use at Johnson Space Center.

Source record↗
NASA NTRSDate not supplied

TPSAS-NF1676L-33992-DND

The CERES Science Team integrates and fuses observations from 6 CERES instruments aboard the Terra, Aqua, S-NPP, and NOAA-20 missions with data from more than 20 other unique data sources. Following the November 2017 launch of CERES Flight Model 6 (FM6) onboard NOAA20, CERES has now amassed over 80 instrument-years of valuable Earth radiation budget data. The rapidly growing volume of CERES data coupled with the introduction of new data products alongside improvements to existing science algorithms fosters the requirement for faster, more flexible, and scalable data production and orchestration. New virtualized, cloud-centric compute hardware hosted by the NASA Langley Research Center’s (LaRC) Atmospheric Sciences Data Center (ASDC) provides an ideal environment for these ever-increasing data production demands for CERES. This poster discusses updates to the implementation of the CERES Data Management Team’s (DMT) CERES AuTomAted job Loading sYSTem (CATALYST), a custom data processing workflow engine for CERES, to use on-demand computing resources to perform automated CERES data production processing in a Linux-based container environment. Linux containers provide CERES the flexibility to build multiple production environments in containers tailored for specific workloads and allow effortless provisioning of resources based on the CERES Science Team’s data production requirements.

Thomas N. Hillyer↗
DOE OSTI2022

Nuclear Quality Assurance Certification of the MCNP-ORIGEN Activation Automation tool

The MCNP-ORIGEN Activation Automation (MOAA) tool couples MCNP and ORIGEN-S, allowing to support ATR and TREAT experiments by predicting their composition and activity before their irradiation takes place. MOAA is developed and maintained by the Irradiation Experiment Neutronics Analysis group (C-150) at INL. This poster presents the motivation behind the development of MOAA, its main workflow, as well as the software quality assurance (SQA) efforts required by LWP-13620, "Managing Information Technology Assets."

07 ISOTOPE AND RADIATION SOURCES↗
DOE OSTI2019

Risk-based monitoring designs for detecting CO 2 leakage through abandoned wellbores: An application of NRAP’s WLAT and DREAM tools

We report that one suggested approach to managing the risk of geologic CO 2 storage projects is to perform risk-based subsurface monitoring targeting early leak detection. The challenge in risk-based monitoring is that leakage impacts are expected to be small in comparison with the footprint of the stored CO 2 plume and could be at depths ranging from the depth of storage (often ~1 km bgs) up to surficial aquifers. Addressing this challenge requires a multidisciplinary effort capable of characterizing potential subsurface leakage in terms of likelihood, location, magnitude, impacts to overlying aquifers, and uncertainty as well as innovative monitoring technology design and implementation. The National Risk Assessment Partnership (NRAP) was established to develop related risk assessment tools to provide guidance on such assessments. Here we demonstrate the application workflow of two of the NRAP computational tools, WLAT (Wellbore Leakage Analysis Tool) and DREAM (Designs for Risk Evaluation and Management), to a hypothetical CO 2 storage site based on a study area in the Midwestern United States. This workflow shows how CO 2 leakage may be estimated, evaluated, and monitored in terms of risk. Results provide guidance on monitoring configurations and technologies which minimize the time to leak detection, maximize the probability detection, and provide cost considerations.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2013

Implementation of an Integrated Information Management System for the US DOE Hanford Tank Farms Project

In its role as the Tank Operations Contractor at the U.S. Department of Energy's site in Hanford, WA, Washington River Protection Solutions, LLC is implementing an integrated document control and configuration management system. This system will combine equipment data with technical document data that currently resides in separate disconnected databases. The new system will provide integrated information, enabling users to more readily identify the documents that relate to a structure, system, or component and vice-versa. Additionally, the new system will automate engineering work processes through electronic workflows, and where practical and feasible provide integration with design authoring tools. Implementation of this system will improve configuration management of the technical baseline, increase work process efficiencies, support the efficient design of future large projects, and provide a platform for the efficient future turnover of technical baseline data and information.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2016

A Semi-Automated Workflow Solution for Data Set Publication

In order to address the need for published data, considerable effort has gone into formalizing the process of data publication. From funding agencies to publishers, data publication has rapidly become a requirement. Digital Object Identifiers (DOI) and data citations have enhanced the integration and availability of data. The challenge facing data publishers now is to deal with the increased number of publishable data products and most importantly the difficulties of publishing diverse data products into an online archive. The Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC), a NASA-funded data center, faces these challenges as it deals with data products created by individual investigators. This paper summarizes the challenges of curating data and provides a summary of a workflow solution that ORNL DAAC researcher and technical staffs have created to deal with publication of the diverse data products. Finally, the workflow solution presented here is generic and can be applied to data from any scientific domain and data located at any data center.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2015

Pooling the resources of the CMS Tier-1 sites

The CMS experiment at the LHC relies on 7 Tier-1 centres of the WLCG to perform the majority of its bulk processing activity, and to archive its data. During the first run of the LHC, these two functions were tightly coupled as each Tier-1 was constrained to process only the data archived on its hierarchical storage. This lack of flexibility in the assignment of processing workflows occasionally resulted in uneven resource utilisation and in an increased latency in the delivery of the results to the physics community.The long shutdown of the LHC in 2013-2014 was an opportunity to revisit this mode of operations, disentangling the processing and archive functionalities of the Tier-1 centres. The storage services at the Tier-1s were redeployed breaking the traditional hierarchical model: each site now provides a large disk storage to host input and output data for processing, and an independent tape storage used exclusively for archiving. Movement of data between the tape and disk endpoints is not automated, but triggered externally through the WLCG transfer management systems.With this new setup, CMS operations actively controls at any time which data is available on disk for processing and which data should be sent to archive. Thanks to the high-bandwidth connectivity guaranteed by the LHCOPN, input data can be freely transferred between disk endpoints as needed to take advantage of free CPU, turning the Tier-1s into a large pool of shared resources. The output data can be validated before archiving them permanently, and temporary data formats can be produced without wasting valuable tape resources. Lastly, the data hosted on disk at Tier-1s can now be made available also for user analysis since there is no risk any longer of triggering chaotic staging from tape.In this contribution, we describe the technical solutions adopted for the new disk and tape endpoints at the sites, and we report on the commissioning and scale testing of the service. We detail the procedures implemented by CMS computing operations to actively manage data on disk at Tier-1 sites, and we give examples of the benefits brought to CMS workflows by the additional flexibility of the new system.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2024

Integrating the PanDA Workload Management System with the Vera C. Rubin Observatory

The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three to four days and produce tens of petabytes of raw image data and associated calibration data over the course of the experiment’s run. More than 20 terabytes of data must be stored every night, and annual campaigns to reprocess the entire dataset since the beginning of the survey will be conducted over ten years. The Production and Distributed Analysis (PanDA) system was evaluated by the Rubin Observatory Data Management team and selected to serve the Observatory’s needs due to its demonstrated scalability and flexibility over the years, for its Directed Acyclic Graph (DAG) support, its support for multi-site processing, and its highly scalable complex workflows via the intelligent Data Delivery Service (iDDS). PanDA is also being evaluated for prompt processing where data must be processed within 60 seconds after image capture. This paper will briefly describe the Rubin Data Management system and its Data Facilities (DFs). Finally, it will describe in depth the work performed in order to integrate the PanDA system with the Rubin Observatory to be able to run the Rubin Science Pipelines using PanDA.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Understanding the Impact of Data Staging for Coupled Scientific Workflows

We report the rate of data generated by cutting-edge experimental science facilities and large-scale simulations enabled by current high-performance computing (HPC) systems has continued to grow at a far greater pace than the development of the network and storage capabilities on which these systems rely. To cope with this challenge, scientist are moving toward the creation of autonomous experiments and HPC simulations using machine learning. However, efficiently moving, storing, and processing large amounts of data away from the point of origin presents an incredible challenge. In-memory computing, in situ analysis, data staging, and data streaming are recognized viable alternatives to traditional file-based methods for transferring data between coupled workflows. However, the performance trade-offs and limitations for these methods are not fully understood when used in HPC applications. This article presents a comprehensive performance assessment of the current solutions for data staging when applied to applications that are not necessary I/O intensive which makes them not ideal candidates for these methods. Our study is based on experiments running at scale on Oak Ridge National Laboratory's Summit supercomputer using applications and simulations that cover typical computational motifs and patterns. We investigated the usability and cost/benefit trade-offs of staging algorithms for HPC applications under different scenarios and highlight opportunities for optimizing the dataflow between coupled simulation workflows.

97 MATHEMATICS AND COMPUTING↗