Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

A galactic approach to neutron scattering science

Neutron scattering science is leading to significant advances in our understanding of materials and will be key to solving many of the challenges that society is facing today. Improvements in scientific instruments are actually making it more difficult to analyze and interpret the results of experiments due to the vast increases in the volume and complexity of data being produced and the associated computational requirements for processing that data. New approaches to enable scientists to leverage computational resources are required, and Oak Ridge National Laboratory (ORNL) has been at the forefront of developing these technologies. We recently completed the design and initial implementation of a neutrons data interpretation platform that allows seamless access to the computational resources provided by ORNL. For the first time, we have demonstrated that this platform can be used for advanced data analysis of correlated quantum materials by utilizing the world's most powerful computer system, Frontier. In particular, we have shown the end-to-end execution of the DCA++ code to determine the dynamic magnetic spin susceptibility χ(q, ω) for a single-band Hubbard model with Coulomb repulsion U/t = 8 in units of the nearest-neighbor hopping amplitude t and an electron density of n = 0.65. The following work describes the architecture, design, and implementation of the platform and how we constructed a correlated quantum materials analysis workflow to demonstrate the viability of this system to produce scientific results.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Real-World Experiences Adopting Workflows at Exascale on the ExaAM Project

The purpose of this study is to discuss the experiential lessons associated with adopting scientific workflows in the Exascale Additive Manufacturing project (ExaAM) through the lens of Perceived Characteristic of Innovation (PCI). Besides the implementation, the factors we considered critical to the adoption of the workflow are provenance, sustainable automation, implementation challenges, and integration/compatibility challenges. Through conversations and interviews among the program managers, project leads, and software engineers, we have developed critical insight and strategies to overcome the obstacles and augment the successful adoption and long-term use of these workflows in ExaAM and beyond. We hope our work will pave the way for others in the research community to develop and use workflows in their respective science domains.

Malviya, Addi Thakur↗
DOE OSTI2021

Waste Compliance and Tracking System (WCATS) Version 3 Requirements Document

This document describes the end-user requirements for the Waste Compliance and Tracking System (WCATS) project in accordance with the WCATS Software Quality Management Plan, EPC-WMP-WCATSPLAN-001. The WCATS application shall support the generation, characterization, processing, and shipment of LANL radioactive, hazardous, and industrial waste. Regulatory drivers include RCRA hazardous waste, DOT shipping, NNSA nuclear material control and accountability, DOE nuclear safety, TSDF permit, and transuranic waste certification requirements. The system will utilize a task-based architecture that supports the spectrum of treatment, storage, disposal, administrative, and characterization based unit operations necessary to manage waste from cradle to grave. The application design shall readily accommodate new facilities, processes, workflow, signature requirements, and so forth, via end-user established metadata. WCATS will provide support for representing waste storage and disposal facilities, buildings, rooms, and grid layouts (x, y, z) to support waste and radioactive material inventory management. Nuclear material at risk (MAR), DOE hazard rating (e.g., Category II facility) compliance per DOE-STD-1027, and permit inventory requirements will be configurable for any storage or disposal facility, or waste operation, and the system will automatically evaluate and enforce those requirements. In addition, the application will support the characterization and management of the entire range of hazardous and radioactive wastes (TRU, MTRU, LLW, MLLW, hazardous waste, etc.) that might be colocated or processed at a permitted facility. Some capabilities not found in traditional systems include user-defined tank systems for liquid waste, user-defined work paths (i.e., sequence of operations), and an equipment subsystem for tracking the calibration, maintenance, and inspection of tools used to process waste, such as torque wrenches, scales, pH probes, etc. The application incorporates a desktop and mobile user interface as shown in Figure 1. The mobile interface supports field operations, such as waste item characterization, intra-facility transfers, internal and external audits, and shipment preparation and receipt.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2015

Data Transfer Study HPSS Archiving

The movement of the large amounts of data produced by codes run in a High Performance Computing (HPC) environment can be a bottleneck for project workflows. To balance filesystem capacity and performance requirements, HPC centers enforce data management policies to purge old files to make room for new computation and analysis results. Users at Oak Ridge Leadership Computing Facility (OLCF) and many other HPC user facilities must archive data to avoid data loss during purges, therefore the time associated with data movement for archiving is something that all users must consider. This study observed the difference in transfer speed from the originating location on the Lustre filesystem to the more permanent High Performance Storage System (HPSS). The tests were done with a number of different transfer methods for files that spanned a variety of sizes and compositions that reflect OLCF user data. This data will be used to help users of Titan and other Cray supercomputers plan their workflow and data transfers so that they are most efficient for their project. We will also discuss best practice for maintaining data at shared user facilities.

Big Data↗
DOE OSTI2018

The BayREN Integrated Commercial Retrofits (BRICR) Project: An Introduction and Preliminary Results

BayREN Integrated Commercial Retrofits (BRICR) is a DOE-funded project which aims to enhance the capacity of energy efficiency programs to recruit participants, develop retrofits, and measure outcomes in small and medium-sized commercial buildings, a sector notoriously hard to reach and expensive to serve, that accounts for 2/3 of US commercial floor space. To address these barriers, BRICR leverages existing incentives, financing, data, and open source software to facilitate two paths for comprehensive improvements: a deep energy retrofit, or serial upgrades integrated into capital improvement and maintenance cycles. BRICR is developing an integrated workflow for iterative energy modeling of commercial buildings for city energy program managers and auditors - starting with mass building-scale simulation based on public records and proceeding through audit, retrofit, and measurement and verification stages. BRICR builds on existing tools including LBNL’s CBES, NREL’s OpenStudio, PNNL’s Audit Template, and DOE’s BuildingSync and SEED PlatformTM. At each stage, BRICR uses available information to inform simulations (starting with public records but iteratively augmented with observations from energy program staff) to improve the quality of the models that inform decision making. This paper presents initial results from energy models of 1699 office and retail buildings in San Francisco. Building stock data from public records were translated to create a BuildingSync file for each building which was stored in SEED. Each BuildingSync file was then translated to multiple OpenStudio Workflow files for EnergyPlus simulation to estimate energy savings of energy conservation measures (ECM). Energy savings predictions for each ECM were written back to an updated BuildingSync file for each building and re-uploaded to SEED. The distribution of baseline energy simulations was calibrated against the publicly disclosed distribution of energy benchmarking data to increase confidence in results.

Piette, M↗
DOE OSTI2016

Data Provenance Hybridization Supporting Extreme-Scale Scientific WorkflowApplications

As high performance computing (HPC) infrastructures continue to grow in capability and complexity, so do the applications that they serve. HPC and distributed-area computing (DAC) (e.g. grid and cloud) users are looking increasingly toward workflow solutions to orchestrate their complex application coupling, pre- and post-processing needs To gain insight and a more quantitative understanding of a workflow’s performance our method includes not only the capture of traditional provenance information, but also the capture and integration of system environment metrics helping to give context and explanation for a workflow’s execution. In this paper, we describe IPPD’s provenance management solution (ProvEn) and its hybrid data store combining both of these data provenance perspectives.

provenance↗
DOE OSTI2025

Automating Traffic Microsimulation from SYNCHRO UTDF to SUMO

Modern transportation research relies on seamlessly integrating traffic signal data with robust network representation and simulation tools. This study presents utdf2gmns, an open-source Python tool that automates conversion of the Universal Traffic Data Format, including network representation, signalized intersections, and turning volumes into the General Modeling Network Specification (GMNS) Standard. The resulting GMNS-compliant network can be converted for microsimulation in SUMO. By automatically extracting intersection control parameters and aligning them with GMNS conventions, utdf2gmns minimizes manual preprocessing and data loss. utdf2gmns also integrates with the Sigma-X engine to extract and visualize key traffic control metrics, such as phasing diagrams, turning volumes, volume-tocapacity ratios, and control delays. This streamlined workflow enables efficient scenario testing, accurate model building, and consistent data management. Validated through case studies, utdf2gmns reliably models complex urban corridors, promoting reproducibility and standardization. Documentation is available on GitHub and PyPI, supporting easy integration and community engagement.

Luo, Roy [ORNL] (ORCID:0009000312909983)↗
DOE OSTI2014

F3D Image Processing and Analysis for Many - and Multi-core Platforms

F3D is written in OpenCL, so it achieve[sic] platform-portable parallelism on modern mutli-core CPUs and many-core GPUs. The interface and mechanims to access F3D core are written in Java as a plugin for Fiji/ImageJ to deliver several key image-processing algorithms necessary to remove artifacts from micro-tomography data. The algorithms consist of data parallel aware filters that can efficiently utilizes[sic] resources and can work on out of core datasets and scale efficiently across multiple accelerators. Optimizing for data parallel filters, streaming out of core datasets, and efficient resource and memory and data managements over complex execution sequence of filters greatly expedites any scientific workflow with image processing requirements. F3D performs several different types of 3D image processing operations, such as non-linear filtering using bilateral filtering and/or median filtering and/or morphological operators (MM). F3D gray-level MM operators are one-pass constant time methods that can perform morphological transformations with a line-structuring element oriented in discrete directions. Additionally, MM operators can be applied to gray-scale images, and consist of two parts: (a) a reference shape or structuring element, which is translated over the image, and (b) a mechanism, or operation, that defines the comparisons to be performed between the image and the structuring element. This tool provides a critical component within many complex pipelines such as those for performing automated segmentation of image stacks. F3D is also called a "descendent" of Quant-CT, another software we developed in the past. These two modules are to be integrated in a next version. Further details were reported in: D.M. Ushizima, T. Perciano, H. Krishnan, B. Loring, H. Bale, D. Parkinson, and J. Sethian. Structure recognition from high-resolution images of ceramic composites. IEEE International Conference on Big Data, October 2014.

Source record↗
DOE OSTI2025

Machine learning methods for weather forecasting

SAND2025-14466O This repository contains code for developing, training, and evaluating machine learning models for weather and climate forecasting, including forecast skill assessment, feature importance analysis, and reproducible workflows for model comparison. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Holthuijzen, Maike [Sandia National Lab. (SNL-CA),↗
NASA NTRS2001

Enabling Advanced Automation in Spacecraft Operations with the Spacecraft Emergency Response System

True autonomy is the Holy Grail of spacecraft mission operations. The goal of launching a satellite and letting it manage itself throughout its useful life is a worthy one. With true autonomy, the cost of mission operations would be reduced to a negligible amount. Under full autonomy, any problems (no matter the severity or type) that may arise with the spacecraft would be handled without any human intervention via some combination of smart sensors, on-board intelligence, and/or smart automated ground system. Until the day that complete autonomy is practical and affordable to deploy, incremental steps of deploying ever-increasing levels of automation (computerization of once manual tasks) on the ground and on the spacecraft are gradually decreasing the cost of mission operations. For example, NASA's Goddard Space Flight Center (NASA-GSFC) has been flying spacecraft with low cost operations for several years. NASA-GSFC's SMEX (Small Explorer) and MIDEX (Middle Explorer) missions have effectively deployed significant amounts of automation to enable the missions to fly predominately in 'light-out' mode. Under light-out operations the ground system is run without human intervention. Various tools perform many of the tasks previously performed by the human operators. One of the major issues in reducing human staff in favor of automation is the perceived increased in risk of losing data, or even losing a spacecraft, because of anomalous conditions that may occur when there is no one in the control center. When things go wrong, missions deploying advanced automation need to be sure that anomalous conditions are detected and that key personal are notified in a timely manner so that on-call team members can react to those conditions. To ensure the health and safety of its lights-out missions, NASA-GSFC's Advanced Automation and Autonomy branch (Code 588) developed the Spacecraft Emergency Response System (SERS). The SERS is a Web-based collaborative environment that enables secure distributed fault and resource management. The SERS incorporates the use of intelligent agents, threaded discussions, workflow, database connectivity, and links to a variety of communications devices (e.g., two-way paging, PDA's, and Internet phones) via commercial gateways. When the SERS detects a problem, it notifies on-call team members, who then can remotely take any necessary actions to resolve the anomalies.The SERS goes well beyond a simple '911' system that sends out an error code to everyone with a pager. Instead, SERS' software agents send detailed data (i.e., notifications) to the most appropriate team members based on the type and severity of the anomaly and the skills of the on-call team members. The SERS also allows the team members to respond to the notifications from their wireless devices. This unique capability ensures rapid response since the team members no longer have to go to a PC or the control center for every anomalous event. Most importantly, the SERS enables safe experimentation with various techniques for increasing levels of automation, leading to robust autonomy. For the MIDEX missions at NASA GSFC, the SERS is used to provide 'human-in-the-loop' automation. During lights-out operations, as greater control is given to the MIDEX automated systems, the SERS can be configured to page remote personnel and keep them informed regarding actions taking place in the control center. Remote off-duty operators can even be given the option of enabling or inhibiting a specific automated response in near real time via their two-way pagers. The SERS facilitates insertion of new technology to increase automation, while maintaining the safety and security of mission resources. This paper will focus on SERS' overall functionality and how SERS has been designed to handle the monitoring and emergency response for missions with varying levels of automation. The paper will also convey some of the key lessons learned from SERS' deployment across of variety of missions, highlighting this incremental approach to achieving 'robust autonomy'.

Breed, Julie↗
NASA NTRS2014

Exploration Technologies for Operations

Although the International Space Station (ISS) assembly has been completed, the Operations support teams continue to seek more efficient and effective ways to prepare for and conduct the ISS operations and future exploration missions beyond low earth orbit. This search for improvement has led to a significant collaboration between the NASA research and advanced software development community at NASA Ames Research Center and the Mission Operations community at NASA Johnson Space Center. Since 2001, NASA Ames Research Center has been developing and applying its advanced intelligent systems and human systems integration research to mission operations tools for several of the unmanned Mars missions operations. Since 2006, NASA Ames Research Center has also been developing and applying its advanced intelligent systems and human systems integration research to mission operations tools for manned operations support with the Mission Operations Directorate at NASA Johnson Space Center. This paper discusses the completion of the development and deployment of a variety of intelligent and human systems technologies adopted for manned mission operations. The technologies associated with the projects include advanced software systems for operations and human-centered computing. Human-centered computing looks to the processes and procedures that people do to perform any given job, then attempts to identify opportunities to improve these processes and procedures. In particular, for mission operations, improvements are quantified by specifically identifying how a tool can increase a persons efficiency, enhance a persons functional capability, andor improve the assurance of a persons decisions. The Ames development team has collaborated with the Mission Operations team to identify areas of efficiencies through technology infusion applications in support of the Plan, Train, and Fly activities of human-spaceflight mission operations. The specific applications discussed in this paper are in the areas of mission planning systems, mission operations design modeling and workflow automation, advanced systems monitoring, mission control technologies, search tools, training management tools, spacecraft solar array management, spacecraft power management, and spacecraft attitude planning. We discuss these specific projects between the Ames Research Center and the Johnson Space Centers Mission Operations Directorate, and how these technologies and projects are enhancing the mission operations support for the International Space Station. We also discuss the challenges, problems, and successes associated with long-distance and multi-year development projects between the research team at Ames and the Mission Operations customers at Johnson Space center. Finally, we discuss how these technology infusion applications and underlying technologies might be used in the future to support on-board operations of the crew and spacecraft systems as human exploration expands beyond low earth orbit to destinations in the solar system where communications delays will require more on-board autonomy and planning by the crew. Longer communications delays will require that the ground mission operations support will be primarily strategic in nature, while the tactical level of planning, systems monitoring and control, and failure analysisisolationrecovery will be the responsibility of both the spacecraft autonomous systems and the crew. Our expectation is that the technologies

mission operations↗
DOE OSTI2014

F3D Image Processing and Analysis for Many - and Multi-core Platforms

F3D is written in OpenCL, so it achieve[sic] platform-portable parallelism on modern mutli-core CPUs and many-core GPUs. The interface and mechanims to access F3D core are written in Java as a plugin for Fiji/ImageJ to deliver several key image-processing algorithms necessary to remove artifacts from micro-tomography data. The algorithms consist of data parallel aware filters that can efficiently utilizes[sic] resources and can work on out of core datasets and scale efficiently across multiple accelerators. Optimizing for data parallel filters, streaming out of core datasets, and efficient resource and memory and data managements over complex execution sequence of filters greatly expedites any scientific workflow with image processing requirements. F3D performs several different types of 3D image processing operations, such as non-linear filtering using bilateral filtering and/or median filtering and/or morphological operators (MM). F3D gray-level MM operators are one-pass constant time methods that can perform morphological transformations with a line-structuring element oriented in discrete directions. Additionally, MM operators can be applied to gray-scale images, and consist of two parts: (a) a reference shape or structuring element, which is translated over the image, and (b) a mechanism, or operation, that defines the comparisons to be performed between the image and the structuring element. This tool provides a critical component within many complex pipelines such as those for performing automated segmentation of image stacks. F3D is also called a "descendent" of Quant-CT, another software we developed in the past. These two modules are to be integrated in a next version. Further details were reported in: D.M. Ushizima, T. Perciano, H. Krishnan, B. Loring, H. Bale, D. Parkinson, and J. Sethian. Structure recognition from high-resolution images of ceramic composites. IEEE International Conference on Big Data, October 2014.

Source record↗
DOE OSTI2016

The FIFE Project at Fermilab

The FabrIc for Frontier Experiments (FIFE) project is an initiative within the Fermilab Scientific Computing Division designed to steer the computing model for non-LHC Fermilab experiments across multiple physics areas. FIFE is a collaborative effort between experimenters and computing professionals to design and develop integrated computing models for experiments of varying size, needs, and infrastructure. The major focus of the FIFE project is the development, deployment, and integration of solutions for high throughput computing, data management, database access and collaboration management within an experiment. To accomplish this goal, FIFE has developed workflows that utilize Open Science Grid compute sites along with dedicated and commercial cloud resources. The FIFE project has made significant progress integrating into experiment computing operations several services including a common job submission service, software and reference data distribution through CVMFS repositories, flexible and robust data transfer clients, and access to opportunistic resources on the Open Science Grid. The progress with current experiments and plans for expansion with additional projects will be discussed. FIFE has taken the leading role in defining the computing model for Fermilab experiments, aided in the design of experiments beyond those hosted at Fermilab, and will continue to define the future direction of high throughput computing for future physics experiments worldwide.

Box, D.↗
DOE OSTI2025

EnergyPlus-MCP: A model-context-protocol server for ai-driven building energy modeling

Traditional building energy modeling with the EnergyPlus building performance simulation engine requires domain expertise, programming skills, and intensive manual efforts limiting its effective adoption. This paper introduces EnergyPlus-MCP, the first open-source Model Context Protocol (MCP) server specifically designed for EnergyPlus simulation workflows, establishing a new foundational infrastructure for AI-driven building energy modeling. The MCP server implements a layered architecture with 35 specialized tools spanning model management, editing and analysis, HVAC and other systems configuration inspection, and simulation execution, enabling Large Language Models to interact with EnergyPlus through conversational interfaces. The server addresses critical workflow barriers by automating model validation, streamlining energy efficiency measures modification, and providing intelligent output management with interactive visualization. Through practical demonstrations using a multi-zone building retrofit analysis, we show how the EnergyPlus-MCP server significantly reduces manual efforts while maintaining full simulation rigor. By providing accessible natural language interfaces to sophisticated building energy analysis, this approach enables scalable deployment of simulation expertise across public and private organizations, educational institutions, and research teams, fundamentally transforming traditional building energy modeling practices.

AI↗
DOE OSTI2020

Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid

Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s world-class high performance computing (HPC) facilities, and (2) our short-term efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows. We propose a more disruptive domain-specific solution: the No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.

Godoy, William↗
DOE OSTI2022

Co-scheduling Ensembles of In Situ Workflows

Molecular dynamics (MD) simulations are widely used to study large-scale molecular systems. HPC systems are ideal platforms to run these studies, however, reaching the necessary simulation timescale to detect rare processes is challenging, even with modern supercomputers. To overcome the timescale limitation, the simulation of a long MD trajectory is replaced by multiple short-range simulations that are executed simultaneously in an ensemble of simulations. Analyses are usually co-scheduled with these simulations to efficiently process large volumes of data generated by the simulations at runtime, thanks to in situ techniques. Executing a workflow ensemble of simulations and their in situ analyses requires efficient co- scheduling strategies and sophisticated management of computational resources so that they are not slowing down each other. In this paper, we propose an efficient method to co-schedule simulations and in situ analyses such that the makespan of the workflow ensemble is minimized. We present a novel approach to allocate resources for a workflow ensemble under resource constraints by using a theoretical framework modeling the workflow ensemble’s execution. We evaluate the proposed approach using an accurate simulator based on the WRENCH simulation framework on various workflow ensemble configurations. Results demonstrate the significance of co-scheduling simulations and in situ analyses that couple data together to benefit from data locality, in which inefficient scheduling decisions can lead to slowdown in makespan up to a factor of 30.

Do, Tu Mai Anh↗
DOE OSTI2020

Mesoscale Science Data Analytics

This is software that will be used to do data analytics in experimental workflows for x-ray mesoscale science. This tool set will provide a mechanism for supporting experiments in many ways, from collecting calibration information and raw data, to managing and viewing data to extracting crystallographic and physical parameters. It will eventually include development of a fully automated workflow that will include statistical information and prediction capabilities to support the scientists in decision-making and replanning their experiments when necessary.

Sweeney, Christine↗
DOE OSTI2017

Argo NodeOS: Toward Unified Resource Management for Exascale

Exascale systems are expected to feature hundreds of thousands of compute nodes with hundreds of hardware threads and complex memory hierarchies with a mix of on-package and persistent memory modules. In this context, the Argo project is developing a new operating system for exascale machines. Targeting production workloads using workflows or coupled codes, we improve the Linux kernel on several fronts. We extend the memory management of Linux to be able to subdivide NUMA memory nodes, allowing better resource partitioning among processes running on the same node. We also add support for memory-mapped access to node- local, PCIe-attached NVRAM devices and introduce a new scheduling class targeted at parallel runtimes supporting user- level load balancing.

Perarnau, Swann↗