Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 685 records · Page 38
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Instantiation of the Damara Tern Platform for Advanced Materials and Manufacturing Technologies (AMMT) Program Collaborative Data Management

This work package focused on deploying an instance of the Damara Tern platform to support AMMT collaborative research activities. The objectives were to provide selected AMMT collaborators with access to a shared environment for capturing operations, trackables, and associated metadata, and to implement data entry functionalities that reflect site-specific procedures. Key activities included creating configurable, schema-driven entry forms and validating the data collection process. The report details the deployment process, the platform infrastructure, and the implemented data entry workflows, providing a reference for end users and establishing a foundation for future production-scale deployments.

36 MATERIALS SCIENCE↗
DOE OSTI2016

XSIM Final Report: Modelling the Past and Future of Identity Management for Scientific Collaborations

The eXtreme Science Identity Management (XSIM1) research project: collected and analyzed real world data on virtual organization (VO) identity management (IdM) representing the last 15+ years of collaborative DOE science; constructed a descriptive VO IdM model based on that data; used the model and existing trends to project the direction for IdM in the 2020 timeframe; and provided guidance to scientific collaborations and resource providers that are implementing or seeking to improve IdM functionality. XSIM conducted over 20 semi­structured interviews of representatives from scientific collaborations and resource providers, both in the US and Europe; the interviewees supported diverse set of scientific collaborations and disciplines. We developed a definition of “trust,” a key concept in IdM, to understand how varying trust models affect where IdM functions are performed. The model identifies how key IdM data elements are utilized in collaborative scientific workflows, and it has the flexibility to describe past, present and future trust relationships and IdM implementations. During the funding period, we gave more than two dozen presentations to socialize our work, encourage feedback, and improve the model; we also published four refereed papers. Additionally, we developed, presented, and received favorable feedback on three white papers providing practical advice to collaborations and/or resource providers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Enabling modern data discovery for atmospheric measurements

The Atmospheric Radiation Measurement (ARM) user facility is a US Department of Energy Office of Science user facility that is managed and operated through a collaborative effort led by nine US Department of Energy national laboratories. The ARM Data Center, located at Oak Ridge National Laboratory, is responsible for the timely collection, processing, and delivery of data products to the scientific community. The ARM Data Center holds more than 11,000 data products, including metadata collected from field campaigns, instruments, value-added products, and principal investigator–contributed data. These data sets are checked for successful transfer (for most data, this transfer is carried out automatically via the network; however, some of the largest data sets and some of the most remote sites require manual shipping of hard disks) and both the data and metadata are processed to a standard format, which is an ARM-standardized structure, via the Network Common Data Form. The Network Common Data Form is a self-describing binary format with many compatible software tools. Once processed, the data are cataloged, stored in the ARM Data Archive, and made discoverable through association with an array of metadata-characterizing information, such as location and measurement classification. These metadata enable powerful search capabilities through the ARM Data Center Data Discovery interface. This paper discusses the workflow of how the new discovery system has been redesigned from user requirements and how the data are distributed to the scientific community.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

From Edge to HPC: Investigating Cross-Facility Data Streaming Architectures

In this paper, we investigate three cross-facility data streaming architectures, Direct Streaming (DTS), Proxied Streaming (PRS), and Managed Service Streaming (MSS). We examine their architectural variations in data flow paths and deployment feasibility, and detail their implementation using the Data Streaming to HPC (DS2HPC) architectural framework and the SciStream memory-to-memory streaming toolkit on the production-grade Advanced Computing Ecosystem (ACE) infrastructure at Oak Ridge Leadership Computing Facility (OLCF). We present a workflow-specific evaluation of these architectures using three synthetic workloads derived from the streaming characteristics of scientific workflows. Through simulated experiments, we measure streaming throughput, round-trip time, and overhead under work sharing, work sharing with feedback, and broadcast and gather messaging patterns commonly found in AI-HPC communication motifs. Our study shows that DTS offers a minimal-hop path, resulting in higher throughput and lower latency, whereas MSS provides greater deployment feasibility and scalability across multiple users but incurs significant overhead. PRS lies in between, offering a scalable architecture whose performance matches DTS in most cases.

George, Anjus [ORNL] (ORCID:0000000179737061)↗
DOE OSTI2024

Rethinking Programming Paradigms in the QC-HPC Context

Programming for today’s quantum computers is making significant strides toward modern workflows compatible with high performance computing (HPC), but fundamental challenges still remain in the integration of these vastly different technologies. Quantum computing (QC) programming languages share some common ground, as well as their emerging runtimes and algorithmic modalities. In this short paper, we explore avenues of refinement for the quantum processing unit (QPU) in the context of many-tasks management, asynchronous or otherwise, in order to understand the value it can play in linking QC with HPC. Through examples, we illustrate how its potential for scientific discovery might be realized.

Wong, Elaine↗
DOE OSTI2024

MFiX Development Updates

This presentation discusses recent developments of the Multiphase Flow with Interphase eXchanges (MFiX) software. A brief review of all modeling approaches along with their cost versus accuracy is provided to guide users when selecting a model for a given application. The main new features of the past five official releases of MFiX are described. Improvement in chemistry management allow for easier simulation setup, and faster simulation speed. Progress in the implementation of a thin-wall boundary condition is presented. The major new model development over the past year is the release of the Glued Sphere Particle model (GSP), where component spheres are combined together to represent non-spherical particles. The integration of Machine Learning (ML) workflow in the CFD process is discussed with two applications: a surrogate model for stiff chemistry and the development of a PIC stress model using ML.

Dietiker, Jeff↗
DOE OSTI2020

Mapping Vegetation at Species Level with High-Resolution Multispectral and Lidar Data Over a Large Spatial Area: A Case Study with Kudzu

Mapping vegetation species is critical to facilitate related quantitative assessment, and mapping invasive plants is important to enhance monitoring and management activities. Integrating high-resolution multispectral remote-sensing (RS) images and lidar (light detection and ranging) point clouds can provide robust features for vegetation mapping. However, using multiple sources of high-resolution RS data for vegetation mapping on a large spatial scale can be both computationally and sampling intensive. Here, we designed a two-step classification workflow to potentially decrease computational cost and sampling effort and to increase classification accuracy by integrating multispectral and lidar data in order to derive spectral, textural, and structural features for mapping target vegetation species. We used this workflow to classify kudzu, an aggressive invasive vine, in the entire Knox County (1362 km2) of Tennessee (U.S.). Object-based image analysis was conducted in the workflow. The first-step classification used 320 kudzu samples and extensive, coarsely labeled samples (based on national land cover) to generate an overprediction map of kudzu using random forest (RF). For the second step, 350 samples were randomly extracted from the overpredicted kudzu and labeled manually for the final prediction using RF and support vector machine (SVM). Computationally intensive features were only used for the second-step classification. SVM had constantly better accuracy than RF, and the producer’s accuracy, user’s accuracy, and Kappa for the SVM model on kudzu were 0.94, 0.96, and 0.90, respectively. SVM predicted 1010 kudzu patches covering 1.29 km2 in Knox County. We found the sample size of kudzu used for algorithm training impacted the accuracy and number of kudzu predicted. The proposed workflow could also improve sampling efficiency and specificity. Our workflow had much higher accuracy than the traditional method conducted in this research, and could be easily implemented to map kudzu in other regions as well as map other vegetation species.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

From bench to biofactory: high-throughput technologies and automated workflows to accelerate biomanufacturing

Microbial production of target molecules has advanced significantly in recent years driven by innovations in enzyme engineering, DNA synthesis, and genomic editing. However, to access the massive potential of microbial production, a vast parametric space remains to be investigated to optimize these biobased processes for a robust bioeconomy. Here, we review the current state of the art, some key challenges and possible solutions. We see a critical role of automation, high-throughput technologies, self-driving and cloud labs, and data management to enable Artificial Intelligence/Machine Learning and mechanistic models to overcome the design space challenges and accelerate the development of novel bio-based solutions. Accurate models will expedite the development and scale-up of engineered microbes for a range of final products from many starting materials.

Petzold, Christopher J↗
DOE OSTI2015

SU-E-I-58: Experiences in Setting Up An Online Fluoroscopy Tracking System in a Large Healthcare System

Purpose: Transitioning from a paper based to an online system for tracking fluoroscopic case information required by state regulation and to conform to NCRP patient dose tracking suggestions. Methods: State regulations require documentation of operator, equipment, and some metric of tube output for fluoroscopy exams. This information was previously collected in paper logs, which was cumbersome and inefficient for the large number of fluoroscopic units across multiple locations within the system. The “tech notes” feature within Siemens’ Syngo workflow RIS was utilized to create an entry form for technologists to input case information, which was sent to a third party vendor for archiving and display though an online web based portal. Results: Over 55k cases were logged in the first year of implementation, with approximately 6,500 cases per month once fully online. A system was built for area managers to oversee and correct data, which has increased the accuracy of inputted values. A high-dose report was built to automatically send notifications when patients exceed trigger levels. In addition to meeting regulatory requirements, the new system allows for larger scale QC in fluoroscopic cases by allowing comparison of data from specific procedures, locations, equipment, and operators so that instances that fall outside of reference levels can be identified for further evaluation. The system has also drastically improved identification of operators without documented equipment specific training. Conclusion: The transition to online fluoroscopy logs has improved efficiency in meeting state regulatory requirements as well as allowed for identification of particular procedures, equipment, and operators in need of additional attention in order to optimize patient and personnel doses, while high dose alerts improve patient care and follow up. Future efforts are focused on incorporating case information from outside of radiology, as well as on automating processes for increased efficiencies.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2013

Tracking Provenance in ORNL's Flexible Research Platforms

Provenance is dened as information about the origin of objects, a concept that applies to both physical and digital objects and often overlaps both. The use of provenance in systems designed for research is an important but forgotten feature. Provenance allows for proper and exact tracking of information, its use, its lineage, its derivations and other metadata that are important for correctly adhering to the scien- tic method. In our project's prescribed use of provenance, researchers can determine detailed information about the use of sensor data in their experiments on ORNL's Flexible Research Platforms (FRPs). Our project's provenance system, Provenance Data Management System (ProvDMS), tracks information starting with the creation of information by an FRP sensor. The system determines station information, sensor information, and sensor channel information. The system allows researchers to derive generations of experiments from the sensor data and tracks their hierarchical flow. Key points can be seen in the history of the information as part of the information's workflow. The concept of provenance and its usage in science is relatively new and while used in other cases around the world, our project's provenance diers in a key area. To keep track of provenance, most systems must be designed or redesigned around the new provenance system. Our system is designed as a cohesive but sepa- rate entity and allows for researchers to continue using their own methods of analysis without being constrained in their ways in order to track the provenance. We have designed ProvDMS using a lightweight provenance library, Core Provenance Library (CPL) v.6 In addition to keeping track of sensor data experiments and its provenance, ProvDMS also provides a web-enabled visualization of the inheritance.

provenance↗
DOE OSTI2022

Developing Fuel Cell Electric Powertrain Architectures for Commercial Vehicles

Here, this article addresses the architecture development for a commercial vehicle fuel cell electric powertrain by establishing a clear multi-step formalized workflow that employs a unique technoeconomic solution for architecture selection. The power capability of the fuel cell, the energy capacity and chemistry of the electrical energy storage (battery), the DC-DC converter (including the input current rating and isolation resistance requirements), the traction drive solution, the on-board hydrogen storage solution, and the real-time power-split management of the fuel cell and the battery are all considered and developed in this effort. The methods were used to select architecture for Class 8 urban, regional, and line haul applications. When compared to traditional load-following power-split controllers, an energy management power-split controller can increase system energy efficiency by up to 19.5%. The energy-efficient power-split controller may increase the required battery capacity for an equivalent life by up to 2.6 times. The impact on the total cost of ownership (TCO) for a variety of financial cases demonstrates that high C-rate capable batteries have the potential to provide better TCO solutions over a six-year vehicle life than low C-rate capable batteries. To achieve TCO parity with the 600 A non-isolated DC-DC converter case, the specific choice of the fuel cell DC-DC converter to achieve a target power output based on current levels (from 500 A to 2400 A) shows that efficiency decreases and cost increases due to the higher current, requiring fuel cell prices to decrease by $50–$100/kW, $60–$110/kW, and $100–$220/kW for urban, regional, and line haul applications, respectively. Key recommendations for powertrain system architectures are provided, with specifics based on vehicle dynamics, mission and application characteristics, end customer use-case profile, critical powertrain component costs, and architecture selection cost function. This study rigorously demonstrates the interplay of the above parameters, with a focus on TCO, and provides application decision-makers with a mechanism and well-defined set of impact factors to consider as part of their architecture selection process.

25 ENERGY STORAGE↗
NASA NTRS2010

NASA SensorWeb and OGC Standards for Disaster Management

I. Goal: Enable user to cost-effectively find and create customized data products to help manage disasters; a) On-demand; b) Low cost and non-specialized tools such as Google Earth and browsers; c) Access via open network but with sufficient security. II. Use standards to interface various sensors and resultant data: a) Wrap sensors in Open Geospatial Consortium (OGC) standards; b) Wrap data processing algorithms and servers with OGC standards c) Use standardized workflows to orchestrate and script the creation of these data; products. III. Target Web 2.0 mass market: a) Make it simple and easy to use; b) Leverage new capabilities and tools that are emerging; c) Improve speed and responsiveness.

Mandl, Dan↗
DOE OSTI2016

Evaluation of Graph Pattern Matching Workloads in Graph Analysis Systems

Graph analysis has emerged as a powerful method for data scientists to represent, integrate, query, and explore heterogeneous data sources. As a result, graph data management and mining became a popular area of research, and led to the development of plethora of systems in recent years. Unfortunately, the number of emerging graph analysis systems and the wide range of applications, coupled with a lack of apples-to-apples comparisons, make it difficult to understand the trade-offs between different systems and the graph operations for which they are designed. A fair comparison of these systems is a challenging task for the following reasons: multiple data models, non-standardized serialization formats, various query interfaces to users, and diverse environments they operate in. To address these key challenges, in this paper we present a new benchmark suite by extending the Lehigh University Benchmark (LUBM) to cover the most common capabilities of various graph analysis systems. We provide the design process of the benchmark, which generalizes the workflow for data scientists to conduct the desired graph analysis on different graph analysis systems. Equipped with this extended benchmark suite, we present performance comparison for nine subgraph pattern retrieval operations over six graph analysis systems, namely NetworkX, Neo4j, Jena, Titan, GraphX, and uRiKA. Through the proposed benchmark suite, this study reveals both quantitative and qualitative findings in (1) implications in loading data into each system; (2) challenges in describing graph patterns for each query interface; and (3) different sensitivity of each system to query selectivity. We envision that this study will pave the road for: (i) data scientists to select the suitable graph analysis systems, and (ii) data management system designers to advance graph analysis systems.

Hong, Seokyong↗
DOE OSTI2021

Generalizable coordination of large multiscale workflows: challenges and learnings at scale

The advancement of machine learning techniques and the heterogeneous architectures of most current supercomputers are propelling the demand for large multiscale simulations that can automatically and autonomously couple diverse components and map them to relevant resources to solve complex problems at multiple scales. Nevertheless, despite the recent progress in workflow technologies, current capabilities are limited to coupling two scales. In the first-ever demonstration of using three scales of resolution, we present a scalable and generalizable framework that couples pairs of models using machine learning and in situ feedback. We expand upon the massively parallel Multiscale Machine-Learned Modeling Infrastructure (MuMMI), a recent, award-winning workflow, and generalize the framework beyond its original design. We discuss the challenges and learnings in executing a massive multiscale simulation campaign that utilized over 600,000 node hours on Summit and achieved more than 98% GPU occupancy for more than 83% of the time. We present innovations to enable several orders of magnitude scaling, including simultaneously coordinating 24,000 jobs, and managing several TBs of new data per day and over a billion files in total. Finally, we describe the generalizability of our framework and, with an upcoming open-source release, discuss how the presented framework may be used for new applications.

Bhatia, Harsh↗
NASA NTRS2011

The Challenges of Releasing Human Data for Analysis

The NASA Johnson Space Center s (NASA JSC) Committee for the Protection of Human Subjects (CPHS) recently approved the formation of two human data repositories: the Lifetime Surveillance of Astronaut Health Repository (LSAH-R) for clinical data and the Life Sciences Data Archive Repository (LSDA-R) for research data. The establishment of these repositories forms the foundation for the release of data and information beyond the scope for which the data was originally collected. The release of clinical and research data and information is primarily managed by two NASA groups: the Evidence Base Working Group (EBWG), consisting of members of both repositories, and the LSAH Policy Board. The goal of unifying these repositories and their processes is to provide a mutually supportive approach to handling medical and research data, to enhance the use of medical and research data to reduce risk, and to promote the understanding of space physiology, countermeasures and other mitigation strategies. Over the past year, both repositories have received over 100 data and information requests from a wide variety of requesters. The disposition of these requests has highlighted the challenges faced when attempting to make data collected on a unique set of subjects available beyond the original intent for which the data were collected. As the EBWG works through each request, many considerations must be factored into account when deciding what data can be shared and how - from the Privacy Act of 1974 and the Health Insurance Portability and Accountability Act (HIPAA), to NASA s Health Information Management System (10HIMS) and Human Experimental and Research Data Records (10HERD) access requirements. Additional considerations include the presence of the data in the repositories and vetting requesters for legitimacy of their use of the data. Additionally, fair access must be ensured for intramural, as well as extramural investigators. All of this must be considered in the formulation of the charters, policies and workflows for the human data repositories at NASA.

Fitts, Mary↗
NASA NTRS2022

2020 I.F._Wickizer_New CE Tool for MDC_Final Report

Our team surveyed available products and found no readily-available/U.S. commercial or Agency products which supported our envisioned workflow for the Mission Design Center and the concurrent engineering (CE) process we use. Therefore, we sought to create our own CE tool. Changes in agency policy during the performance period allowed us to shift direction and instead adapt our previous legacy tool (Atlas) to include the collaboration features we sorely needed. Atlas relied upon databases for its back-end data management. Poseidon, the work proposed under this effort, was also intended to leverage from that previous back end (in part). However, in May 2021, EUSO announced that “SBU/CUI Information Can Now Be Shared/Stored Within O365 Without Encryption.” Consequently, we made the decision to use O365 in place of our back end. This change provides for version history, collaborative simultaneous editing, and other collaboration tools agency-wide that were previously unavailable with the old database architecture. The new Atlas O365 has more flexibility and capability for users. It’s now easier for users to switch between using the tool for concurrent engineering and individual subsystem engineering. A version was delivered for concurrent engineering of small satellite missions; so far, it has been used successfully on the Aeolus MDC study. Atlas O365 will facilitate the design and assessment of Small Satellite Missions at low Concept Maturity Level at Ames. Feasibility assessments on mission concepts still at a low CML permit strategic planning and decision-making efforts at the center level about which concepts should be pursued and proposed.

concurrent engineering↗
DOE OSTI2017

Virtual machine provisioning, code management, and data movement design for the Fermilab HEPCloud Facility

The Fermilab HEPCloud Facility Project has as its goal to extend the current Fermilab facility interface to provide transparent access to disparate resources including commercial and community clouds, grid federations, and HPC centers. This facility enables experiments to perform the full spectrum of computing tasks, including data-intensive simulation and reconstruction. We have evaluated the use of the commercial cloud to provide elasticity to respond to peaks of demand without overprovisioning local resources. Full scale data-intensive workflows have been successfully completed on Amazon Web Services for two High Energy Physics Experiments, CMS and NOνA, at the scale of 58000 simultaneous cores. This paper describes the significant improvements that were made to the virtual machine provisioning system, code caching system, and data movement system to accomplish this work. The virtual image provisioning and contextualization service was extended to multiple AWS regions, and to support experiment-specific data configurations. A prototype Decision Engine was written to determine the optimal availability zone and instance type to run on, minimizing cost and job interruptions. We have deployed a scalable on-demand caching service to deliver code and database information to jobs running on the commercial cloud. It uses the frontiersquid server and CERN VM File System (CVMFS) clients on EC2 instances and utilizes various services provided by AWS to build the infrastructure (stack). We discuss the architecture and load testing benchmarks on the squid servers. We also describe various approaches that were evaluated to transport experimental data to and from the cloud, and the optimal solutions that were used for the bulk of the data transport. Finally, we summarize lessons learned from this scale test, and our future plans to expand and improve the Fermilab HEP Cloud Facility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗