Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “distributed services”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Joint Genome Institute Analysis Workflow Service (JAWS) v2.7

The U.S. Department of Energy Joint Genome Institute (JGI) has developed the JGI Analysis Workflow Service (JAWS) as a distributed framework to run computational workflows across diverse high-performance computing (HPC) and cloud environments. JAWS enhances the reusability, scalability, and robustness of scientific workflows by orchestrating data movement, code execution, and results retrieval across multiple DOE facilities. At its core, JAWS integrates the Cromwell workflow engine to run workflows expressed in the Workflow Description Language (WDL), ensuring portability and interoperability. To provide consistent runtime environments, JAWS employs container technologies such as Shifter, Apptainer, and Docker. Workflow tasks are managed via HTCondor on HPC backends, while Globus ensures secure and efficient data transfer between sites. JAWS is deployed as a multi-site workflow manager across national laboratory computing facilities, with dedicated instances supporting community projects such as the National Microbiome Data Collaborative (NMDC) and KBase. This distributed, service-oriented architecture enables users to "write once, run anywhere," providing scalable, production-quality workflow execution.

Kirton, Edward↗
DOE OSTI2021

Resilient Operation of Power Distribution Systems Using MPC-Based Critical Service Restoration: Preprint

Power distribution systems are more prone to disruptions and cause most power system outages. We propose a service restoration technique to recover the system service (electricity delivery) following an extreme event-triggered substation outage. The proposed technique considers the problem of controlling distributed energy resources (DERs) of a distribution system with the objective of achieving maximum load pick up while satisfying network flow and voltage constraints. The problem is formulated as a model predictive control (MPC), where a linearized optimal power flow (OPF) model is employed to describe the network. The formulation is augmented with a ramping (up) reserve product for the DERs to ensure an upward monotonic load restoration as time evolves. We perform simulations considering the IEEE 13-bus test feeder integrated with wind, solar, microturbine, and energy storage devices. We demonstrate the efficacy of the devised technique in restoring the system loads monotonically, without shedding previously restored loads. We also show the benefit of co-optimization of power and reserve products for DERs on service restoration. In addition, the capability of the technique in regulating nodal voltages and reducing renewable power curtailment is demonstrated.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2023

Distributed Grid-Sensing Service Layer: A System Architecture Blueprint

In this report, a scalable, decentralized, sensor-oriented communication blueprint is introduced. The proposed architecture has been specifically designed to support the mass deployment of sensors and also enable data consumers to access the underlying data streams using a distributed systems approach. With the proposed approach, sensors and data clients can be decoupled from the physical communication infrastructure and migrated into a modern, software-defined infrastructure ecosystem that can be configured to suit the end application’s requirements. In its current iteration, this blueprint defines the interactions, processes, and expected outcomes that each individual component within the architecture must support the overall ecosystem’s needs. Within this work, such an assembly of services is called the data bus; it represents the heart of the distributed grid sensing services architecture, a multiyear project that aims to design, implement, and test a reference sensor-data dissemination network that can support the needs of the extended grid state. We expect this blueprint to guide our future implementation efforts

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Congestion Management Solutions for Enhanced Distribution System Operations with Aggregated Distribution Grid Resources Providing Grid Services and Market Participation

Microgrids and other aggregations of distribution grid resources (DGRs) are poised to actively participate in electricity markets and provide essential grid services in the coming years. In fact, DGRs already play such a role through behind-the-meter (BTM) demand response programs and small-scale BTM dispatchable generation initiatives. At the same time, the rapid growth of artificial intelligence (AI) and cryptocurrency datacenters imposes significant, often unpredictable, demands on the power distribution system. Aggregated DGRs can serve as flexible resources that help mitigate these pressures by using available transmission and distribution capacity more efficiently, supporting resource adequacy and other reliability services, and providing bridge strategies while long-term transmission infrastructure is being developed. The impacts this activity will have on distribution networks are not fully understood and could present significant challenges for distribution utilities due to capacity constraints and the need for congestion management. Technical issues include reverse power flow, variability and possible degradation of equipment integrity, voltage violations, and customer power quality concerns. These issues will likely intensify as electricity market operators across the United States implement Federal Energy Regulatory Commission Order 2222 over the next few years.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS2015

A Web Service and Android Application for the Distribution of Rainfall Estimates and Earth Observation Data

The full potential of Satellite Rainfall Estimates (SRE) can only be realized if timely access to the datasets is possible. Existing data distribution web portals are often focused on global products and offer limited customization options, especially for the purpose of routine regional monitoring. Furthermore, most online systems are designed to meet the needs of desktop users, limiting the compatibility with mobile devices. In response to the growing demand for SRE and to address the current limitations of available web portals a project was devised to create a set of freely available applications and services, available at a common portal that can: (1) simplify cross-platform access to Tropical Rainfall Measuring Mission Online Visualization and Analysis System (TOVAS) data (including from Android mobile devices), (2) provide customized and continuous monitoring of SRE in response to user demands and (3) combine data from different online data distribution services, including rainfall estimates, river gauge measurements or imagery from Earth Observation missions at a single portal, known as the Tropical Rainfall Measuring Mission (TRMM) Explorer. The TRMM Explorer project suite includes a Python-based web service and Android applications capable of providing SRE and ancillary data in different intuitive formats with the focus on regional and continuous analysis. The outputs include dynamic plots, tables and data files that can also be used to feed downstream applications and services. A case study in Southern Angola is used to describe the potential of the TRMM Explorer for SRE distribution and analysis in the context of ungauged watersheds. The development of a collection of data distribution instances helped to validate the concept and identify the limitations of the program, in a real context and based on user feedback. The TRMM Explorer can successfully supplement existing web portals distributing SRE and provide a cost-efficient resource to small and medium-sized organizations with specific SRE monitoring needs, namely in developing and transition countries.

precipitation↗
DOE OSTI2020

Customizable Scale-Out Key-Value Stores

Enterprise KV stores are often not well suited for HPC applications, and thus cumbersome end-to-end KV design customization is required to meet the needs of modern HPC applications. To this end, in this article we present bespoKV, an adaptive, extensible, and scale-out KV store framework. bespoKV decouples the KV store design into the control plane for distributed management and the data plane for local data store. For the control plane, bespoKVprovides pre-built modules, called controlets, supporting common distributed functionalities (e.g., replication, consistency, and topology) and their various combinations. This decoupling allows bespoKV to take a user-provided single-server KV store, called a datalet, and transparently enables a scalable and fault-tolerant distributed KV store service. The resulting distributed stores are also adaptive to consistency or topology requirement changes and can be easily extended for new types of services. Such specializations enable innovative uses of KV stores in HPC applications, especially for emerging applications that utilize KV-friendly workloads. We evaluate bespoKV in a local testbed as well as in a public cloud settings. Experiments show that bespoKV-enabled distributed KV stores scale horizontally to a large number of nodes, and performs comparably and sometimes 1.2× to 2.6× better than the state-of-the-art systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

ATLAS Data Analysis using a Parallel Workflow on Distributed Cloud-based Services with GPUs

A new type of parallel workflow is developed for the ATLAS experiment at the Large Hadron Collider, that makes use of distributed computing combined with a cloud-based infrastructure. This has been developed for a specific type of analysis using ATLAS data, one popularly referred to as Simulation-Based Inference (SBI). The JAX library is used for the parts of the workflow to compute gradients as well as accelerate program execution using just-in-time compilation, which becomes essential in a full SBI analysis and can also offer significant speed-ups in more traditional types of analysis.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

Experimental Analysis of Distribution Network Voltage Regulation Using Smart Inverters

Smart inverters (SIs) have demonstrated their potential to provide grid services for both transmission and distribution systems. One of these grid services, distribution network voltage regulation by SIs, has the potential to improve network voltage regulation through controlling the reactive and active power output of the SIs. Voltage regulation by SIs will be distributed and might be better suited to controlling local conditions to complement traditional voltage-regulating assets, e.g., tap-changing transformers, capacitor banks, and line voltage regulators. There is a gap in the literature on comparing the SI response characteristics when the SIs are controlled by a local controller or external control signals. This paper presents an experimental study to characterize SI reactive power regulation responses to two different control methods: autonomous control and remote dispatch. We found that SI reactive power regulation responses exhibit important differences between these methods in terms of delays and ramp rate. Finally, power-hardware-in-the-loop (PHIL) tests were conducted to evaluate the performance of these two methods. The PHIL test results show that the SI response characteristics for autonomous control and remote dispatch need to be considered when planning for distribution network voltage regulation using SIs.

autonomous control↗
DOE OSTI2020

Large-Scale Adoption of Self-Synchronized Universal Droop Controller-Based Inverters to Enable Ancillary Services for Different Modes of Distribution System Operations

This work proposes the large-scale adoption of self-synchronized universal droop controller (SUDC)-based inverters to enable ancillary services for different modes of distribution system operations. The IEEE 123 bus system was modeled on a real-time simulator to study the performance of large-scale adoption of SUDC inverters in a distribution system. The resulting data collected shows that the voltage and the frequency were regulated within ranges, such as less than 5% for voltage and less than 0.5% for frequency, under different load variations and grid operations. Also, the black start was achieved within 0.4 s without any voltage overshoot. Through the simulation and validation on a small microgrid and the IEEE 123 bus distribution system, it can be concluded that the SUDC was successfully adopted to regulate the voltage and the frequency within the given ranges, and black start achieved within 1 s without voltage overshoot for different modes of distribution system operations.

14 SOLAR ENERGY↗
DOE OSTI2022

Automatic DDoS Attack Detection on SDNs: Preprint

Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks pose a serious threat to computing networks - especially to critical systems within the U.S. electrical grid. As attack mechanisms have increased in complexity and variety, more sophisticated detection mechanisms have become necessary to ensure network security. This paper explores the use of artificial intelligence to automate the process of detection and mitigation of DoS and DDoS attacks within the framework of Software-Defined Networking (SDN), to a high degree. Machine learning algorithms are trained to recognize DoS and DDoS attacks and are deployed in real-time to mitigate malicious network traffic. The results show a well-tuned gradient-boosted decision tree detecting DoS and DDoS attacks, as well as initial successful mitigation of attacks within an SDN framework.

cyber detection↗
NASA NTRS1989

Standard services for the capture, processing, and distribution of packetized telemetry data

Standard functional services for the capture, processing, and distribution of packetized data are discussed with particular reference to the future implementation of packet processing systems, such as those for the Space Station Freedom. The major functions are listed under the following major categories: input processing, packet processing, and output processing. A functional block diagram of a packet data processing facility is presented, showing the distribution of the various processing functions as well as the primary data flow through the facility.

Stallings, William H.↗
DOE OSTI2020

Mochi: Composing Data Services for High-Performance Computing Environments

Technology enhancements and the growing breadth of application workflows running on high-performance computing (HPC) platforms drive the development of new data services that provide high performance on these new platforms, provide capable and productive interfaces and abstractions for a variety of applications, and are readily adapted when new technologies are deployed. The Mochi framework enables composition of specialized distributed data services from a collection of connectable modules and subservices. Rather than forcing all applications to use a one-size-fits-all data staging and I/O software configuration, Mochi allows each application to use a data service specialized to its needs and access patterns. This paper introduces the Mochi framework and methodology. The Mochi core components and microservices are described. Examples of the application of the Mochi methodology to the development of four specialized services are detailed. Finally, a performance evaluation of a Mochi core component, a Mochi microservice, and a composed service providing an object model is performed. The paper concludes by positioning Mochi relative to related work in the HPC space and indicating directions for future work.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2004

Grist : grid-based data mining for astronomy

The Grist project is developing a grid-technology based system as a research environment for astronomy with massive and complex datasets. This knowledge extraction system will consist of a library of distributed grid services controlled by a workflow system, compliant with standards emerging from the grid computing, web services, and virtual observatory communities. This new technology is being used to find high redshift quasars, study peculiar variable objects, search for transients in real time, and fit SDSS QSO spectra to measure black hole masses. Grist services are also a component of the 'hyperatlas' project to serve high-resolution multi-wavelength imagery over the Internet. In support of these science and outreach objectives, the Grist framework will provide the enabling fabric to tie together distributed grid services in the areas of data access, federation, mining, subsetting, source extraction, image mosaicking, statistics, and visualization.

grid computing↗
NASA NTRS1980

Gas distribution equipment in hydrogen service - Phase II

The hydrogen permeability of three different types of commercially available natural gas polyethylene pipes was determined. Ring tensile tests were conducted on permeability-exposed and as-received samples. Hydrogen-methane leakage experiments were also performed. The results show no selective leakage of hydrogen via Poiseuille, turbulent, or orifice flow (through leaks) on the distribution of blends of hydrogen and methane. The data collected show that the polyethylene pipe is 4 to 6 times more permeable to hydrogen than to methane.

Jasionowski, W. J.↗
NASA NTRS2003

Data Access Tools And Services At The Goddard Distributed Active Archive Center (GDAAC)

As one of the largest providers of Earth Science data from the Earth Observing System, GDAAC provides the latest data from the Moderate Resolution Imaging Spectroradiometer (MODIS), Atmospheric Infrared Sounder (AIRS), Solar Radiation and Climate Experiment (SORCE) data products via GDAAC's data pool (50TB of disk cache). In order to make this huge volume of data more accessible to the public and science communities, the GDAAC offers multiple data access tools and services: Open Source Project for Network Data Access Protocol (OPeNDAP), Grid Analysis and Display System (GrADS/DODS) (GDS), Live Access Server (LAS), OpenGlS Web Map Server (WMS) and Near Archive Data Mining (NADM). The objective is to assist users in retrieving electronically a smaller, usable portion of data for further analysis. The OPeNDAP server, formerly known as the Distributed Oceanographic Data System (DODS), allows the user to retrieve data without worrying about the data format. OPeNDAP is capable of server-side subsetting of HDF, HDF-EOS, netCDF, JGOFS, ASCII, DSP, FITS and binary data formats. The GrADS/DODS server is capable of serving the same data formats as OPeNDAP. GDS has an additional feature of server-side analysis. Users can analyze the data on the server there by decreasing the computational load on their client's system. The LAS is a flexible server that allows user to graphically visualize data on the fly, to request different file formats and to compare variables from distributed locations. Users of LAS have options to use other available graphics viewers such as IDL, Matlab or GrADS. WMS is based on the OPeNDAP for serving geospatial information. WMS supports OpenGlS protocol to provide data in GIs-friendly formats for analysis and visualization. NADM is another access to the GDAAC's data pool. NADM gives users the capability to use a browser to upload their C, FORTRAN or IDL algorithms, test the algorithms, and mine data in the data pool. With NADM, the GDAAC provides an environment physically close to the data source. NADM will benefit users with mining or offer data reduction algorithms by reducing large volumes of data before transmission over the network to the user.

Pham, Long↗
DOE OSTI2024

A Managed Tokens Service for Securely Keeping and Distributing Grid Tokens

Fermilab is transitioning authentication and authorization for grid operations to using bearer tokens based on the WLCG Common JWT (JSON Web Token) Profile. One of the functionalities that Fermilab experimenters rely on is the ability to automate batch job submission, which in turn depends on the ability to securely refresh and distribute the necessary credentials to experiment job submit points. Thus, with the transition to using tokens for grid operations, we needed to create a service that would obtain, refresh, and distribute tokens for experimenters’ use. This service would avoid the need for experimenters to be experts in obtaining their own tokens and would better protect the most sensitive long-lived credentials. Further, the service needed to be widely scalable, as we are currently keeping credentials active for approximately 15 experiments, each with 1-3 different credentials, and distributing those credentials to 2-20 submit points per experiment, with those numbers steadily increasing. To address these issues, we created and deployed a Managed Tokens service. The service is written in Go, taking advantage of that language’s native concurrency primitives to easily be able to scale operations as we onboard experiments. The service uses as its first credentials a set of kerberos keytabs, stored on the same secure machine that the Managed Tokens service runs on. These kerberos credentials allow the service to use htgettoken via condor_vault_storer to store vault tokens in the HTCondor credential managers (credds) that run on the batch system scheduler machines (HTCondor schedds); as well as downloading a local, shorter-lived copy of the vault token. The kerberos credentials are then also used to distribute copies of the locally-stored vault tokens to experiment submit points. When experimenters schedule jobs to be submitted, these distributed vault tokens are used to access a Hashicorp Vault instance (run separately from the Managed Tokens service), and previously-stored refresh tokens there are used to obtain the bearer token that is submitted with the job. We will discuss here the design of the Managed Tokens service, including elaborating on certain choices we made with regards to concurrent operations, configuration, monitoring, and deployment.

Bhat, Shreyas↗
DOE OSTI2021

A Load Switching Group based Feeder-level Microgrid Energy Management Algorithm for Service Restoration in Power Distribution System

This paper presents a load switching group based energy management system (LSG-EMS) for operating microgrids on a distribution feeder powered by one or multiple grid-forming distributed energy resources. Loads on a distribution feeder are divided into load switching groups that can be remotely switched on and off. The LSG-EMS algorithm, formulated as a mixed-integer linear programming (MILP) problem, has an objective function of maximizing the served loads while minimizing the total number of switching actions. A new set of topology constraints are developed for allowing multiple microgrids to be formed on the feeder and selecting the optimal supply path. Customer comfort is accounted for by maximizing the supply duration in the customer preferred service period and enforcing a minimum service duration. The proposed method is demonstrated on a modified IEEE 33-bus system using actual customer data. Simulation results show that the LSG-EMS successfully coordinates multiple grid-forming sources by selecting an optimal supply topology that maximizes the supply period of both the critical and noncritical loads while minimizing customer service interruptions in the service restoration process.

Hu, Rongxing↗
DOE OSTI2025

A Managed Tokens Service for Securely Keeping and Distributing Grid Tokens

Fermilab is transitioning authentication and authorization for grid operations to using bearer tokens based on the WLCG Common JWT (JSON Web Token) Profile. One of the functionalities that Fermilab experimenters rely on is the ability to automate batch job submission, which in turn depends on the ability to securely refresh and distribute the necessary credentials to experiment job submit points. Thus, with the transition to using tokens for grid operations, we needed to create a service that would obtain, refresh, and distribute tokens for experimenters' use. This service would avoid the need for experimenters to be experts in obtaining their own tokens and would better protect the most sensitive long-lived credentials. Further, the service needed to be widely scalable, as Fermilab hosts many experiments, each of which would need their own credentials. To address these issues, we created and deployed a Managed Tokens Service. The service is written in Go, taking advantage of that language's native concurrency primitives to easily be able to scale operations as we onboard experiments. The service uses as its first credentials a set of kerberos keytabs, stored on the same secure machine that the Managed Tokens service runs on. These kerberos credentials allow the service to use htgettoken via condor_vault_storer to store vault tokens in the HTCondor credential managers (credds) that run on the batch system scheduler machines (HTCondor schedds); as well as downloading a local, shorter-lived copy of the vault token. The kerberos credentials are then also used to distribute copies of the locally-stored vault tokens to experiment submit points.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗