Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “microservices”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

33 records · Page 2

Cyote-attack Chain Estimator

Attack Chain Estimator (ACE) Application Overview The Attack Chain Estimator (ACE) Application is a sophisticated tool designed for the ingestion, classification, sequencing, and enrichment of cybersecurity threat reports. This application leverages advanced machine learning models and extensive historical data to provide comprehensive insights into cyber threats, specifically targeting Industrial Control Systems (ICS). Purpose The primary functions of the ACE Application include: Ingestion of Cybersecurity Threat Reporting: Capable of ingesting text-based threat reports in markdown or text file format. Supports ingestion of structured data from other sources in STIX/JSON format. Classification of Report’s Text-Based Events: Utilizes a DeBERTa classifier, specifically trained on cybersecurity data, to map the events to MITRE ATT&CK for ICS Tactics and Techniques. Classification is performed using multiple Jupyter notebooks and machine learning workflows hosted as FastAPI microservices: regex_data deberta_base_35_train_hft_classifier_mlflow.ipynb hft_regex_classifier_mlflow.ipynb param_train_hft_classifier_mlflow.ipynb regex_tactic_tech.ipynb Ordering of Tactics, Techniques, and Observable Events: Sequences the identified tactics, techniques, and events to form a coherent attack chain. Enrichment with Historical Attack Chain Details: Enhances the attack chain with details from historical attacks using a Markov model developed from CyOTE Precursor Analysis Report data. The Markov model is available as a FastAPI endpoint for seamless integration. Enrichment with Adversary Emulation Capabilities Data: Integrates adversary emulation capabilities data using MITRE Caldera for OT adversary abilities UUIDs. Export of Output Files: Provides options to export the enriched attack chain in JSON or CSV formats. Routing of Output to Other Applications: Facilitates routing of output to various platforms and applications, including: Threat Intelligence Platforms COREII Scout for Threat Intelligence Analysis COREII Modeling and Simulation for Adversary Emulation Technical Description The ACE Application is an advanced cybersecurity tool designed to provide detailed threat analysis and sequence generation. It is built on a robust architecture that integrates natural language processing, machine learning, and historical data modeling. Key Components: Data Ingestion Module: Handles the input of threat reports and data from various formats, ensuring flexibility in data sources. Classification Engine: Employs DeBERTa-based classifiers hosted as FastAPI microservices to analyze and classify threat report events in accordance with the MITRE ATT&CK framework for ICS. Sequence Generator: Orders the classified events into a logical attack chain, providing clear insight into the sequence of tactics and techniques used in the threat. Enrichment Engine: Integrates historical data and adversary emulation capabilities to enhance the attack chain with valuable context and additional details. The historical data enrichment is powered by a Markov model, which is available as a FastAPI endpoint. Export and Routing Module: Facilitates the export of the enriched attack chain in multiple formats and routes the output to designated applications for further analysis or emulation.

Paul, Tony [Idaho National Laboratory (INL), Idaho↗

The INTERSECT Open Federated Architecture for the Laboratory of the Future

A federated instrument-to-edge-to-center architecture is needed to autonomously collect, transfer, store, process, curate, and archive scientific data and reduce human-in-the-loop needs with (a) common interfaces to leverage community and custom software, (b) pluggability to permit adaptable solutions, reuse, and digital twins, and (c) an open standard to enable adoption by science facilities world-wide. The Selfdriven Experiments for Science/Interconnected Science Ecosystem (INTERSECT) Open Architecture enables science breakthroughs using intelligent networked systems, instruments and facilities with autonomous experiments, “self-driving” laboratories, smart manufacturing and artificial intelligence (AI) driven design, discovery and evaluation. It creates an open federated architecture for the laboratory of the future using a novel approach, consisting of (1) science use case design patterns, (2) a system of systems architecture, and (3) a microservice architecture.

Engelmann, Christian↗

A semantics-driven framework to enable demand flexibility control applications in real buildings

Decarbonising and digitalising the energy sector requires scalable and interoperable Demand Flexibility (DF) applications. Semantic models are promising technologies for achieving these goals, but existing studies focused on DF applications exhibit limitations. These include dependence on bespoke ontologies, lack of computational methods to generate semantic models, ineffective temporal data management and absence of platforms that use these models to easily develop, configure and deploy controls in real buildings. This paper introduces a semantics-driven framework to enable DF control applications in real buildings. The framework supports the generation of semantic models that adhere to Brick and SAREF while using metadata from Building Information Models (BIM) and Building Automation Systems (BAS). The work also introduces a web platform that leverages these models and an actor and microservices architecture to streamline the development, configuration and deployment of DF controls. The paper demonstrates the framework through a case study, illustrating its ability to integrate diverse data sources, execute DF actuation in a real building, and promote modularity for easy reuse, extension, and customisation of applications. The paper also discusses the alignment between Brick and SAREF, the value of leveraging BIM data sources, and the framework's benefits over existing approaches, demonstrating a 75% reduction in effort for developing, configuring, and deploying building controls.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

dCache – Efficient Message Encoding For Inter-Service Communication in dCache: Evaluation of Existing Serialization Protocols as a Replacement for Java Object Serialization

As a well established, large-scale distributed storage system, dCache is used to manage and serve huge amounts of data collected by high energy physics, astrophysics and photon science experiments. Based on a microservices-like architecture, dCache is built as a modular distributed system, where each component provides a different core functionality. These services communicate by passing serialized messages to each other, a core behavior whose performance properties can consequently affect the entire system.This paper compares and evaluates different data serialization protocols in computer science with the objective of replacing and improving upon Java Object Serialization (JOS), which has increasingly presented itself as no longer being sufficiently performant for encoding messages. The criteria for choosing a new framework are collected, analyzed and formalized. The primary motivation for replacing Java serialization for encoding dCache messages is increasing the general speed of message-passing and thereby reducing the round-trip time for user requests. Emphasis is also placed on schema evolution capabilities and framework usability. Approaches for generalizing (de)serialization speed and size measurements based on data structure complexity are introduced, criteria for measuring documentation, learning curve, maintainability and introduction effort are defined. Finally, several selected serialization protocols are evaluated and compared accordingly, concluding with a recommendation for a suitable JOS replacement.

Morschel, Lea↗

Pufferscale: Rescaling HPC Data Services for High Energy Physics Applications

User-space HPC data services are emerging as an appealing alternative to traditional parallel file systems, because of their ability to be tailored to application needs while eliminating unnecessary overheads incurred by POSIX compliance. The High Energy Physics (HEP) community is progressively turning towards such services to enable high-throughput accesses under heavy concurrency to billions of event data produced by instruments and consumed by subsequent analysis workflows. Such services would benefit from the possibility to be rescaled up and down to adapt to changing workloads, as experimental campaigns progress, in order to optimize resource usage. This paper formalizes rescaling a distributed storage system as a multi objective optimization problem considering three criteria: load balance, data balance, and duration of the rescaling operation. We propose a heuristic for rapidly finding a good approximate solution, while allowing users to weight the criteria as needed. The heuristic is evaluated with Pufferscale, a new rescaling manager for microservice-based distributed storage systems. To validate our approach in a real-world ecosystem, we showcase the use of Pufferscale as a means to enable storage malleability in the HEPnOS storage system for HEP applications.

Distributed Storage System↗

An Alternative Timing and Synchronization Approach for Situational Awareness and Predictive Analytics

Accurate and synchronized timing information is required by power system operators for controlling the grid infrastructure (relays, Phasor Measurement Units (PMUs), etc.) and determining asset positions. Satellite-based global positioning system (GPS) is the primary source of timing information. However, GPS disruptions today (both intentional and unintentional) can significantly compromise the reliability and security of our electric grids. A robust alternate source for accurate timing is critical to serve both as a deterrent against malicious attacks and as a redundant system in enhancing the resilience against extreme events that could disrupt the GPS network. To achieve this, we rely on the highly accurate, terrestrial atomic clock-based network for alternative timing and synchronization. In this paper, we discuss an experimental setup for an alternative timing approach. The data obtained from this experimental setup is continuously monitored and analyzed using various time deviation metrics. We also use these metrics to compute deviations of our clock with respect to the National Institute of Standards and Technologys (NIST) GPS data. The results obtained from these metric computations are elaborately discussed. Finally, we discuss the integration of the procedures involved, like real-time data ingestion, metric computation, and result visualization, in a novel microservices-based architecture for situational awareness.

Chinthavali, Supriya↗

Science Use Case Design Patterns for Autonomous Experiments

Connecting scientific instruments and robot-controlled laboratories with computing and data resources at the edge, the Cloud or the high-performance computing (HPC) center enables autonomous experiments, self-driving laboratories, smart manufacturing, and artificial intelligence (AI)-driven design, discovery and evaluation. The Self-driven Experiments for Science / Interconnected Science Ecosystem (INTERSECT) Open Architecture enables science breakthroughs using intelligent networked systems, instruments and facilities with a federated hardware/software architecture for the laboratory of the future. It relies on a novel approach, consisting of (1) science use case design patterns, (2) a system of systems architecture, and (3) a microservice architecture. This paper introduces the science use case design patterns of the INTERSECT Architecture. It describes the overall background, the involved terminology and concepts, and the pattern format and classification. It further offers an overview of the 12 defined patterns and 4 examples of patterns of 2 different pattern classes. It also provides insight into building solutions from these patterns. The target audience are computer, computational, instrument and domain science experts working in the field of autonomous experiments.

Engelmann, Christian↗

The LCLStream Ecosystem for Multi-Institutional Dataset Exploration

We describe a new end-to-end experimental data streaming framework designed from the ground up to support new types of applications – AI training, extremely high-rate X-ray time-of-flight analysis, crystal structure determination with distributed processing, and custom data science applications and visualizers yet to be created. Throughout, we use design choices merging cloud microservices with traditional HPC batch execution models for security and flexibility. This project makes a unique contribution to the DOE Integrated Research Infrastructure (IRI) landscape. By creating a flexible, API-driven data request service, we address a significant need for high-speed data streaming sources for the X-ray science data analysis community. With the combination of data request API, mutual authentication web security framework, job queue system, high-rate data buffer, and complementary nature to facility infrastructure, the LCLStreamer framework has prototyped and implemented several new paradigms critical for future generation experiments.

Rogers, David [ORNL] (ORCID:0000000251871768)↗

interactEM v1.0

An interactive, container-based workflow tool for creating and spawning directed acyclic graphs (DAGs) of operators in a distributed environment. It has a microservices architecture, and flow-based programming model. Current tools like this do not enable streaming of data directly between operators.

Welborn, Sam [Lawrence Berkeley National Laborator↗

Transient Efficiency, Flexibility, and Reliability Optimization of Coal-Fired Power Plants - Final Report

This program developed an advanced model-based monitoring and model-predictive control algorithms for a coal fired power plant (CFPP), and deployed these algorithms in a real-time platform to demonstrate performance benefits for transient flexibility and plant operation efficiency. More specifically, the objectives were successfully achieved through a combination of (i) developing a high-fidelity transient plant model in Apros, which was used as a high-fidelity plant simulation between $100-50\% TMCR$ where TMCR denotes the turbine maximum continuous rating, i.e., baseload, (ii) developing a very fast physics-based reduced-order model (ROM) of the plant, which ran more than $100\times$ faster than real-time, enabling its use as real-time embedded model for model-based estimation (MBE) and model predictive control (MPC) (iii) implementing a real-time MBE based on ROM using a robust unscented Kalman filter (UKF) to continuously tune the ROM to match the measurements from high-fidelity Apros plant model despite significant plant-model mismatch, and thus, obtain a Digital Twin of the plant (iv) designing and implementing a real-time MPC with dual objectives of transient plant load tracking with high ramp rates and minimizing coal consumption, i.e., improving plant efficiency in the baseload-partload operation range of $100-50\% TMCR$. Each key element above was developed and tested individually, and has been reported in corresponding Topical Reports. Finally, all the individual elements were integrated in an overall closed-loop system, that was successfully tested in desktop Simulink test harness simulations with ROM or high-fidelity model as the plant. Thereafter, the Simulink implementation was used to auto-generate C-code and deploy as real-time Docker microservice containers in Linux, and validate that they can run in real-time in the hardware-in-the loop (HIL) setup and produce the same results as in Simulink. The results of the integrated simulation tests in Simulink as well as the real-time HIL deployment are documented in this final report, showing good load tracking for load ramps at $3-4\%/min$ ramp rates, and achieving up to $5.5\%$ reduction in coal relative to baseline operation at $50\% TMCR$ load. The desktop and HIL simulations show successful performance of the overall model based estimation and control solution and achieve the key objectives of the program for flexible, efficient and reliable operation of subcritical coal fired power plants.

20 FOSSIL-FUELED POWER PLANTS↗

INTERSECT Architecture Specification: Use Case Design Patterns (V.0.9)

Connecting scientific instruments and robot-controlled laboratories with computing and data resources at the edge, the Cloud or the high-performance computing (HPC) center enables autonomous experiments, self-driving laboratories, smart manufacturing, and artificial intelligence (AI)-driven design, discovery and evaluation. The Self-driven Experiments for Science / Interconnected Science Ecosystem (INTERSECT) Open Architecture enables science breakthroughs using intelligent networked systems, instruments and facilities with a federated hardware/software architecture for the laboratory of the future. It relies on a novel approach, consisting of (1) science use case design patterns, (2) a system of systems architecture, and (3) a microservice architecture. This document introduces the science use case design patterns of the INTERSECT Architecture. It describes the overall background, the involved terminology and concepts, and the pattern format and classification. It further details the 12 defined patterns and provides insight into building solutions from these patterns. The document also describes the application of these patterns in the context of several INTERSECT autonomous laboratories. The target audience are computer, computational, instrument and domain science experts working in the field of autonomous experiments.

97 MATHEMATICS AND COMPUTING↗

OpenCHAMI Introduction: Cloud-like HPC Management for the Composable future [Slides]

OpenCHAMI is: An independent partnership of large HPC operators and vendor(s); Based on the APIs and Microservices from the Cray System Manager; A set of governance standards for developing HPC system management software aligned with cloud principles; A github organization; A set of software repositories in active development to enable sites to solve their own problems in a shareable way.

97 MATHEMATICS AND COMPUTING↗

INTERSECT Architecture Specification: System-of-Systems Architecture (V.0.9)

Oak Ridge National Laboratory (ORNL)’s Self-driven Experiments for Science / Interconnected Science Ecosystem (INTERSECT) architecture project, titled “An Open Federated Architecture for the Laboratory of the Future”, creates an open federated hardware/software architecture for the laboratory of the future using a novel system of systems (SoS) and microservice architecture approach, connecting scientific instruments, robot-controlled laboratories and edge/center computing/data resources to enable autonomous experiments, “self-driving” laboratories, smart manufacturing, and artificial intelligence (AI)-driven design, discovery and evaluation.

97 MATHEMATICS AND COMPUTING↗

Transient Efficiency, Flexibility, and Reliability Optimization of Coal-Fired Power Plants - Final Program Review

This program developed an advanced model-based monitoring and model-predictive control algorithms for a coal fired power plant (CFPP), and deployed these algorithms in a real-time platform to demonstrate performance benefits for transient flexibility and plant operation efficiency. More specifically, the objectives were successfully achieved through a combination of (i) developing a high-fidelity transient plant model in Apros, which was used as a high-fidelity plant simulation between $100-50\% TMCR$ where TMCR denotes the turbine maximum continuous rating, i.e., baseload, (ii) developing a very fast physics-based reduced-order model (ROM) of the plant, which ran more than $100\times$ faster than real-time, enabling its use as real-time embedded model for model-based estimation (MBE) and model predictive control (MPC) (iii) implementing a real-time MBE based on ROM using a robust unscented Kalman filter (UKF) to continuously tune the ROM to match the measurements from high-fidelity Apros plant model despite significant plant-model mismatch, and thus, obtain a Digital Twin of the plant (iv) designing and implementing a real-time MPC with dual objectives of transient plant load tracking with high ramp rates and minimizing coal consumption, i.e., improving plant efficiency in the baseload-partload operation range of $100-50\% TMCR$. Each key element above was developed and tested individually, and has been reported in corresponding Topical Reports. Finally, all the individual elements were integrated in an overall closed-loop system, that was successfully tested in desktop Simulink test harness simulations with ROM or high-fidelity model as the plant. Thereafter, the Simulink implementation was used to auto-generate C-code and deploy as real-time Docker microservice containers in Linux, and validate that they can run in real-time in the hardware-in-the loop (HIL) setup and produce the same results as in Simulink. The results of the integrated simulation tests in Simulink as well as the real-time HIL deployment are documented in this final report, showing good load tracking for load ramps at $3-4\%/min$ ramp rates, and achieving up to $5.5\%$ reduction in coal relative to baseline operation at $50\% TMCR$ load. The desktop and HIL simulations show successful performance of the overall model based estimation and control solution and achieve the key objectives of the program for flexible, efficient and reliable operation of sub-critical coal fired power plants.

20 FOSSIL-FUELED POWER PLANTS↗

DeepLynx Ecosystem 2025

Poor data integration and governance continue to plague complex engineering projects, resulting in missed cost, schedule, and performance targets. Departments operate in isolated systems with manual data exchange, creating fragmented information that compounds errors and leads to significant delays and cost overruns. The DeepLynx ecosystem addresses these challenges through an open-source, modular data management platform that transforms fragmented project data into an integrated digital thread. Built on a federated microservice architecture, the ecosystem comprises seven specialized tools centered around DeepLynx Nexus, a unified data catalog with hierarchical organization and graph-based navigation capabilities. The ecosystem includes: DeepLynx Stream for real-time timeseries data ingestion from industrial sources; DeepLynx Ingest for governed data uploads with formal review workflows; DeepLynx Lattice for ontology-based entity and relationship extraction; DeepLynx Run for workflow orchestration and secure AI/ML compute; DeepLynx Visualize for 3D digital twin visualization; and DeepLynx Insight for AI-assisted document analysis with traceable, grounded responses. Deployable in cloud, on-premise, or hybrid environments using containerized Docker applications and Helm charts, the DeepLynx ecosystem provides flexible infrastructure that adapts to organizational requirements. By consolidating project data into a unified data lake with role-based access controls and OAuth2 authentication, DeepLynx enables digital thread and digital twin capabilities that improve decision-making, reduce risk, and support complex engineering workflows throughout the project lifecycle.

42 - ENGINEERING↗