Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Alternative Technologies Adopters Outreach Survey Report

Brookhaven National Laboratory (BNL), in collaboration with Ioanna Iliopulos and Christopher Boyd (independent contractors), conducted quantitative and qualitative surveys to document views and opinions of scientists and researchers with firsthand experience operating x-ray devices for a wide range of research applications. The goal of the study was to fill critical information gaps and identify misconceptions widely held among the research community regarding the transition from Cesium-137 irradiators to x-ray devices. The survey instrument focused on three distinct yet interrelated areas that impact users' experience: operational impacts, including infrastructure needs, workflow, personnel management, and training, and regulatory compliance; research design and investigation impacts, including calibration, dosimetry, continuity of previous studies, development of new research agendas, and the outcomes for both; and funding impacts, specifically how grant sources and opportunities have responded to the transition.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2019

BigPanDA: PanDA Workload Management System and its Applications beyond ATLAS

Modern experiments collect peta-scale volumes of data and utilize vast, geographically distributed computing infrastructure that serves thousands of scientists around the world. Requirements for rapid, near real-time data processing, fast analysis cycles and need to run massive detector simulations to support data analysis pose special premium on efficient use of available computational resources. A sophisticated Workload Management System (WMS) is needed to coordinate the distribution and processing of data and jobs in such environment. The ATLAS experiment at CERN uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. While PanDAcurrently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, it runs around 2 million jobs per day on hundreds of Grid sites and serving thousands of ATLAS users. In 2017 about 1.5 exabytes of data were processed with PanDA.In 2012 BigPanDA project project was started with aim to introduce new types of computing resources into ATLAS computing infrastructure, but also to offering PanDA features to different data-intensive applications for projects and experiments outside of ATLAS and High-Energy and Nuclear Physics. In this article we will present accomplishments and discuss possible directions for future work.

Svirin, Pavlo↗
DOE OSTI2015

The Acoustic Emission signal acquired by the microphones placed in the CABRI test device along the fourteen last R.I.A. experiments: an example of reproducible research in nuclear science

The CABRI facility is an experimental nuclear reactor of the French Atomic Energy Commission. It is located at the Cadarache Research Centre in southern France and it is designed to act as a support to the French nuclear infrastructure. The purpose of the new testing programme termed, 'CABRI International Programme' (CIP) is to study the behaviour of PWR-type fuel rods at high burnup, equipped with an 'advanced' cladding, under Reactivity Initiated Accident (RIA) conditions (such as the scenario of a control rod ejection). Within the framework of this programme, piloted and funded by the French Institute of Nuclear Radioprotection and Safety (IRSN), ten tests are to be conducted with a frequency of two tests per year. The LPRE laboratory of the CEA which is in charge of the Preparation, realisation and breakdown of the test results studies the possibility to set up a new test analysis based on the processing of signals coming from sensors placed within the test equipment. During the experimental phase, the behaviour of the fuel element generates acoustic waves which can be detected by two microphones placed upstream and downstream of the test device. Studies showed the interest to realize temporal and spectral analyses on these signals by showing the existence of signatures which can be correlated with physical phenomena as the rod failure or the test shutdown (i.e. scram). The work presented in this article results from the will to consolidate these studies. Since the main phenomenon to be tracked is the fuel rod failure, the aim would be to highlight specific events which would have been precursors of the rod failure in order to use in the future these signals for further interpretation. Such an antecedent works resumption leads to a better understanding of the experimental needs and constitutes a good initial state to prepare the new very fast digital data acquisition systems. Although all the raw data are accessible in the form of text files, analyses and graphics representations were not straightforward to reproduce from the ancient studies since that, on one hand, people who were in charge of the original work left the laboratory and on the other hand because it is not easy when the time passes, even with our own work, to be able to remember the steps of data manipulations and the exact setup: - During the ancient experiments the use of analog data acquisition systems required to digitize tapes to be able to realize computer treatments. That had had for consequence to lose the initial dating. This one must be correctly edited to do temporal comparisons. - Analyses require functions for calculations whose parameters has to be well-known to reach the same results. We thus wished to manage our workflow in the idea that it can be easily reproducible on all the experiments. The object of the work presented in this article was to put in practice this strong bind between the data, treatments and generation of the document in order not to hesitate to do the iteration principle in action. We do not have to be afraid by the data driven analyses. According to the philosophy of the literate programming, the text of the technical document is woven with the computer code that produces all the printed output as tables, graphs for the study eliminating hence the unrealistic cut and paste. This difficulty is not specific to the nuclear domain. For many years, researchers have been worked out solutions to this mundane issue. And, presently, new technologies and high-level programming languages offer us actual answers. We will firstly present the tools applied in our laboratory to implement this workflow, then we will describe the global perception carried out to continue the study of the Acoustic Emission signals recorded by the two microphones during the fourteen last CABRI R.I.A. test.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2015

PANORAMA: An approach to performance modeling and diagnosis of extreme-scale workflows

Here we report that computational science is well established as the third pillar of scientific discovery and is on par with experimentation and theory. However, as we move closer toward the ability to execute exascale calculations and process the ensuing extreme-scale amounts of data produced by both experiments and computations alike, the complexity of managing the compute and data analysis tasks has grown beyond the capabilities of domain scientists. Therefore, workflow management systems are absolutely necessary to ensure current and future scientific discoveries. A key research question for these workflow management systems concerns the performance optimization of complex calculation and data analysis tasks. The central contribution of this article is a description of the PANORAMA approach for modeling and diagnosing the run-time performance of complex scientific workflows. This approach integrates extreme-scale systems testbed experimentation, structured analytical modeling, and parallel systems simulation into a comprehensive workflow framework called Pegasus for understanding and improving the overall performance of complex scientific workflows.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

Godoy, William↗
NASA NTRS2003

Universal Payload Information Management

As the overall manager and integrator of International Space Station (ISS) science payloads, the Payload Operations Integration Center (POIC) at Marshall Space Flight Center has a critical need to provide an information management system for exchange and control of ISS payload files as well as to coordinate ISS payload related operational changes. The POIC's information management system has a fundamental requirement to provide secure operational access not only to users physically located at the POIC, but also to remote experimenters and International Partners physically located in different parts of the world. The Payload Information Management System (PIMS) is a ground-based electronic document configuration management and collaborative workflow system that was built to service the POIC's information management needs. This paper discusses the application components that comprise the PIMS system, the challenges that influenced its design and architecture, and the selected technologies it employs. This paper will also touch on the advantages of the architecture, details of the user interface, and lessons learned along the way to a successful deployment. With PIMS, a sophisticated software solution has been built that is not only universally accessible for POIC customer s information management needs, but also universally adaptable in implementation and application as a generalized information management system.

Elmore, Ralph B.↗
DOE OSTI2021

An intelligent Data Delivery Service for and beyond the ATLAS experiment

The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. iDDS has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service around a workflow-oriented structure to work with existing and emerging use cases in ATLAS and other experiments. Here we will present the motivation for iDDS, its design schema and architecture, use cases and current status, and plans for the future.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Towards a Scalable and Adaptive Application Support Platform for Large-Scale Distributed E-Sciences in High-Performance Network Environments

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models feature diverse scientific workflows as simple as linear pipelines or as complex as a directed acyclic graphs, which must be executed and supported over wide-area networks with massively distributed resources. Application users oftentimes need to manually configure their computing tasks over networks in an ad hoc manner, hence significantly limiting the productivity of scientists and constraining the utilization of resources. The success of these large-scale distributed applications requires a highly adaptive and massively scalable workflow platform that provides automated and optimized computing and networking services. This project is to design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a web-based user interface specially tailored for a target application, a set of user libraries, and several easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in heterogeneous high-performance network environments. SWAMP will enable the automation and management of the entire process of scientific workflows with the convenience of a few mouse clicks while hiding the implementation and technical details from end users. Particularly, we will consider two types of applications with distinct performance requirements: data-centric and service-centric applications. For data-centric applications, the main workflow task involves large-volume data generation, catalog, storage, and movement typically from supercomputers or experimental facilities to a team of geographically distributed users; while for service-centric applications, the main focus of workflow is on data archiving, preprocessing, filtering, synthesis, visualization, and other application-specific analysis. We will conduct a comprehensive comparison of existing workflow systems and choose the best suited one with open-source code, a flexible system structure, and a large user base as the starting point for our development. Based on the chosen system, we will develop and integrate new components including a black box design of computing modules, performance monitoring and prediction, and workflow optimization and reconfiguration, which are missing from existing workflow systems. A modular design for separating specification, execution, and monitoring aspects will be adopted to establish a common generic infrastructure suited for a wide spectrum of science applications. We will further design and develop efficient workflow mapping and scheduling algorithms to optimize the workflow performance in terms of minimum end-to-end delay, maximum frame rate, and highest reliability. We will develop and demonstrate the SWAMP system in a local environment, the grid network, and the 100Gpbs Advanced Network Initiative (ANI) testbed. The demonstration will target scientific applications in climate modeling and high energy physics and the functions to be demonstrated include workflow deployment, execution, steering, and reconfiguration. Throughout the project period, we will work closely with the science communities in the fields of climate modeling and high energy physics including Spallation Neutron Source (SNS) and Large Hadron Collider (LHC) projects to mature the system for production use.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

An intelligent Data Delivery Service for and beyond the ATLAS experiment

The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflows and data management systems, decoupling data pre-processing, delivery, and primary processing in large scale workflows. It is an experiment-agnostic service that has been deployed to serve data carousel (orchestrating efficient processing of tape-resident data), machine learning hyperparameter optimization, active learning, and other complex multi-stage workflows defined via DAG (Directed Acyclic Graph), CWL (Common Workflow Language) and other descriptions, including a growing number of analysis workflows. We will at first introduce some deployed use cases in a summary. Then we will focus on new improvements and use cases under developments in ATLAS, Rubin Observatory and sPHENIX, together with future efforts.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Workflow Provenance in the Computing Continuum for Responsible, Trustworthy, and Energy-Efficient AI

As Artificial Intelligence (AI) becomes more pervasive in our society, it is crucial to develop, deploy, and assess Responsible and Trustworthy AI (RTAI) models, i.e., those that consider not only accuracy but also other aspects, such as explainability, fairness, and energy efficiency. Workflow provenance data have historically enabled critical capabilities towards RTAI. Provenance data derivation paths contribute to responsible workflows through transparency in tracking artifacts and resource consumption. Provenance data are well-known for their trustworthiness helping explainability, reproducibility, and accountability. However, there are complex challenges to achieve RTAI, which are further complicated by the heterogeneous infrastructure in the computing continuum (Edge-Cloud-HPC) used to develop and deploy models. As a result, a significant research and development gap remains between workflow provenance data management and RTAI. In this paper, we present a vision of the pivotal role of workflow provenance in supporting RTAI and discuss related challenges. We present a schematic view between RTAI and provenance, and highlight open research directions.

Santos Souza, Renan↗
DOE OSTI2025

Automating Testing of DUNE Electronics via a Finite State Machine

The Deep Underground Neutrino Experiment (DUNE) is a flagship international collaboration designed to study neutrinos tiny, nearly massless particles that may hold answers to fundamental questions about the Universe. Fermilab s Robotic Test Stand (RTS) plays a critical role in ensuring the quality of approximately 50,000 Application-Specific Integrated Circuit (ASIC) chips that will be used in DUNE s massive liquid argon detectors. These electronics will be inside the cryostat; therefore, they will need to have a high yield of working chips and low noise. To improve the automation and reliability of the RTS, this project focused on designing and implementing a Python-based finite state machine (FSM) to manage chip handling workflows. The FSM was developed as a modular software framework to coordinate robotic arm movements, manage chip tray positions, and monitor system states during testing. Key features include robust error handling routines, a pause/resume system for safe mid-cycle interruptions, and a simulation mode for iterative testing without hardware dependencies. The system was designed to prepare for seamless integration with RTS hardware components such as the robotic arm and vision system. This integration will streamline collaboration and enable efficient deployment of updates across the six total institutions performing testing. The outcomes of this internship contribute to Fermilab s mission to advance high-energy physics and support the DOE s national goals by directly improving the testing of equipment to be used in DUNE. The project also provided valuable experience in software design and contributing to the success of DUNE.

Kang, Caleb [William Rainey Harper Coll.]↗
DOE OSTI2024

Tutorial - Electric Motor and Integrated Traction Drive Thermal Management

The share of vehicles with fully electric propulsion systems is constantly increasing, and so is their traction drive power. The continuous push to increase power of electric vehicle (EV) traction drives necessitates their efficient cooling to prevent damage to temperature sensitive components of the drive system and achieving higher power outputs in a smaller footprint. With increasing power and power density of electric traction drives, their thermal management is becoming increasingly challenging. This tutorial will provide an overview of thermal management approaches for electric motors and power electronics in EV applications. It will review examples of current industry solutions for power-dense electric motor cooling, power electronics (inverter) cooling, their integration concepts and thermal management system solutions. We'll look at the advantages and challenges of power electronics integration into a single traction drive unit and respective thermal management system concepts. We'll talk about barriers to implementation of a unified thermal management system. The tutorial will also review key aspects of thermal management system design: modeling and simulation using FEA and CFD tools, experimental characterization, and general workflow for thermal management system evaluation.

ADVANCED PROPULSION SYSTEMS,DIRECT ENERGY CONVERSI↗
DOE OSTI2025

Automating Testing of DUNE Electronics via a Finite State Machine

The Deep Underground Neutrino Experiment (DUNE) is a flagship international collaboration designed to study neutrinos—tiny, nearly massless particles that may hold answers to fundamental questions about the Universe. Fermilab’s Robotic Test Stand (RTS) plays a critical role in ensuring the quality of approximately 50,000 Application-Specific Integrated Circuit (ASIC) chips that will be used in DUNE’s massive liquid argon detectors. These electronics will be inside the cryostat; therefore, they will need to have a high yield of working chips and low noise. To improve the automation and reliability of the RTS, this project focused on designing and implementing a Python-based finite state machine (FSM) to manage chip handling workflows. The FSM was developed as a modular software framework to coordinate robotic arm movements, manage chip tray positions, and monitor system states during testing. Key features include robust error handling routines, a pause/resume system for safe mid-cycle interruptions, and a simulation mode for iterative testing without hardware dependencies. The system was designed to prepare for seamless integration with RTS hardware components such as the robotic arm and vision system. This integration will streamline collaboration and enable efficient deployment of updates across the six institutions performing testing. The outcomes of this internship contribute to Fermilab’s mission to advance high-energy physics and support the DOE’s national goals by directly improving the testing of equipment to be used in DUNE. The project also provided valuable experience in software design and contributing to the success of DUNE.

Kang, Caleb [Fermilab]↗
NASA NTRS2019

2018 NISAR Applications Workshop: Forest and Disturbance; Workshop Report

Forest lands cover the globe and are important sources for providing ecosystem services including: carbon sequestration, biodiversity, timber, air and water quality. As such, counties around the world have dedicated programs for managing them. Accurate and timely information concerning the status of these forests (moisture, biomass, disturbance type, etc.) is essential to those Nations’ human and ecological health as well as economy. The joint NASA/US Forest Service workshop focused on arming forest land managers with observations and remote sensing information from the upcoming NASA-ISRO (Indian Space Research Organization) SAR (Synthetic Aperture Radar) (NISAR) satellite mission (expected to launch early 2022). Participants included representatives from different US Federal Agencies, private sector, and non-governmental organizations (NGO) that are key players in facilitating integration of Earth Observations (EO) into forest management and decision support workflows. They included scientists, technicians, and program managers with a responsibility for data acquisition and exploitation such as product development, delivery, and use, and capacity building. Discussions were held over two days to convey the broader forest and disturbance community information needs for various representative participants and programs and to facilitate the delivery of NISAR mission geospatial products and observational capabilities. Case studies were presented to demonstrate the current state of practice in the use of SAR remote sensing for applications of direct importance to forest and disturbance land management community. Eleven organizations presented their information requirements in response to a set of questions provided by the NASA team, then the NASA team responded by describing the degree to which NISAR could meet these requirements. Discussion ensued about needed data product specifications to increase utility (e.g., projection, latency, etc.), tools and capacity building. The general findings of this workshop were that (a) NISAR observations will be particularly useful to the global forest carbon and disturbance monitoring applications, but that certain data product design decisions (projections and radiometric and terrain corrections) need to be considered to increase utility; b) the biomass and disturbance detection algorithms meet many of the community needs, however there are other information products of value (e.g., soil moisture or disturbance classification, not just detection) and all products should be compliant with existing community standards for reporting uncertainty; c) providing SAR education to the community will be key specifically thinking about putting the information first and the SAR theory second, providing a simple guide of standard data processing steps (e.g., dB (decibel) to power conversion and speckle filtering); d) the community needs a user-friendly interface for finding free, archived data over their geographic regions of interest; e) user-friendly tools that connect to open-sources GIS (Global Information System) software (e.g., QGIS (Quantum GIS)) that include a graphical user interface (GUI) for SAR processing that enables both download and cloud processing. To integrate these findings and prepare the community before NISAR launches, it was suggested that there be a dedicated NISAR Forest and Disturbance Applications Working Group (as per the specifications in the NISAR Utilization Plan). After launch, it was decided that the community continue capacity building activities.

Stavros, Natasha↗