Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “STORAGE AND NETWORKING SYSTEMS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

A Fixed-Flexible BESS Allocation Scheme for Transmission Networks Considering Uncertainties

Battery energy storage systems (BESSs) can play a key role in mitigating the intermittency and uncertainty associated with adding large amounts of renewable energy to the bulk power system (BPS). Two BESS technologies that have gained prominence in this regard are Lithium-ion (LI) BESS and Vanadium redox flow (VRF) BESS. Here, this paper proposes a fixed-flexible BESS allocation scheme that exploits the complementary characteristics of LI and VRF BESSs to attain optimal techno-economic benefits in a wind-integrated BPS. Studies carried out on relatively large transmission networks demonstrate that benefits such as reduction in system operation cost, wind spillage, voltage fluctuations, and discounted payback period, can be realized by using the proposed scheme.

42 ENGINEERING↗
DOE OSTI2015

An Elegant Sufficiency: Load-Aware Differentiated Scheduling of Data Transfers

We investigate the file transfer scheduling problem, where transfers among different endpoints must be scheduled to maximize pertinent metrics. We propose two new algorithms that exploit the fact that the aggregate bandwidth obtained over a network or at a storage system tends to increase with the number of concurrent transfers—but only up to a certain limit. The first algorithm, SEAL, uses runtime information and data-driven models to approximate system load and adapt transfer schedules and concurrency so as to maximize performance while avoiding saturation. We implement this algorithm using GridFTP as the transfer protocol and evaluate it using real transfer logs in a production WAN environment. Results show that SEAL can improve average slowdowns and turnaround times by up to 25% and worst-case slowdown and turnaround times by up to 50%, compared with the best-performing baseline scheme. Our second algorithm, STEAL, further leverages user-supplied categorization of transfers as either “interactive” (requiring immediate processing) or “batch” (less time-critical). Results show that STEAL reduces the average slowdown of interactive transfers by 63% compared to the best-performing baseline and by 21% compared to SEAL. For batch transfers, compared to the best-performing baseline, STEAL improves by 18% the utilization of the bandwidth unused by interactive transfers. By elegantly ensuring a sufficient, but not excessive, allocation of concurrency to the right transfers, we significantly improve overall performance despite constraints.

Kettimuthu, Rajkumar↗
DOE OSTI2017

When Computers Come First

Ruth Pordes was at Fermilab when computing equipment was first being moved into the laboratory offices. The elevated status of computers at the laboratory was made evident on move-in day.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Understanding and Analyzing Interconnect Errors and Network Congestion on a Large Scale HPC System

Today's High Performance Computing (HPC) systems are capable of delivering performance in the order of petaflops due to the fast computing devices, network interconnect, and back-end storage systems. In particular, interconnect resilience and congestion resolution methods have a major impact on the overall interconnect and application performance. This is especially true for scientific applications running multiple processes on different compute nodes as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks state-of-practice experience reports that detail how different interconnect errors and congestion events occur on large-scale HPC systems. Therefore, in this paper, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors and congestion events. We also study the interaction between interconnect, errors, network congestion and application characteristics.

Kumar, Mohit↗
DOE OSTI2022

Understanding the Impact of Data Staging for Coupled Scientific Workflows

We report the rate of data generated by cutting-edge experimental science facilities and large-scale simulations enabled by current high-performance computing (HPC) systems has continued to grow at a far greater pace than the development of the network and storage capabilities on which these systems rely. To cope with this challenge, scientist are moving toward the creation of autonomous experiments and HPC simulations using machine learning. However, efficiently moving, storing, and processing large amounts of data away from the point of origin presents an incredible challenge. In-memory computing, in situ analysis, data staging, and data streaming are recognized viable alternatives to traditional file-based methods for transferring data between coupled workflows. However, the performance trade-offs and limitations for these methods are not fully understood when used in HPC applications. This article presents a comprehensive performance assessment of the current solutions for data staging when applied to applications that are not necessary I/O intensive which makes them not ideal candidates for these methods. Our study is based on experiments running at scale on Oak Ridge National Laboratory's Summit supercomputer using applications and simulations that cover typical computational motifs and patterns. We investigated the usability and cost/benefit trade-offs of staging algorithms for HPC applications under different scenarios and highlight opportunities for optimizing the dataflow between coupled simulation workflows.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2012

Petascale Data Storage Institute (PDSI) Final Report

Petascale computing infrastructures for scientific discovery make petascale demands on information storage capacity, performance, concurrency, reliability, availability, and manageability. The Petascale Data Storage Institute focuses on the data storage problems found in petascale scientific computing environments, with special attention to community issues such as interoperability, community buy-in, and shared tools. The Petascale Data Storage Institute is a collaboration between researchers at Carnegie Mellon University, National Energy Research Scientific Computing Center, Pacific Northwest National Laboratory, Oak Ridge National Laboratory, Sandia National Laboratory, Los Alamos National Laboratory, University of Michigan, and the University of California at Santa Cruz. Because the Institute focuses on low level files systems and storage systems, its role in improving SciDAC systems was one of supporting application middleware such as data management and system-level performance tuning.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Secure Storage Architectures

The purpose of this report is to clarify the challenges associated with storage for secure enclaves. The major focus areas for the report are: - review of relevant parallel filesystem technologies to identify assets and gaps; - review of filesystem isolation/protection mechanisms, to include native filesystem capabilities and auxiliary/layered techniques; - definition of storage architectures that can be used for customizable compute enclaves (i.e., clarification of use-cases that must be supported for shared storage scenarios); - investigate vendor products related to secure storage. This study provides technical details on the storage and filesystem used for HPC with particular attention on elements that contribute to creating secure storage. We outline the pieces for a a shared storage architecture that balances protection and performance by leveraging the isolation capabilities available in filesystems and virtualization technologies to maintain the integrity of the data. Key Points: There are a few existing and in-progress protection features in Lustre related to secure storage, which are discussed in (Chapter 3.1). These include authentication capabilities like GSSAPI/Kerberos and the in-progress work for GSSAPI/Host-keys. The GPFS filesystem provides native support for encryption, which is not directly available in Lustre. Additionally, GPFS includes authentication/authorization mechanisms for inter-cluster sharing of filesystems (Chapter 3.2). The limitations of key importance for secure storage/filesystems are: (i) restricting sub-tree mounts for parallel filesystem (which is not directly supported in Lustre or GPFS), and (ii) segregation of hosts on the storage network and practical complications with dynamic additions to the storage network, e.g., LNET. A challenge for VM based use cases will be to provide efficient IO forwarding of the parallel filessytem from the host to the guest (VM). There are promising options like para-virtualized filesystems to help with this issue, which are a particular instances of the more general challenge of efficient host/guest IO that is the focus of interfaces like virtio. A collection of bridging technologies have been identified in Chapter 4, which can be helpful to overcome the limitations and challenges of supporting efficient storage for secure enclaves. The synthesis of native filesystem security mechanisms and bridging technologies led to an isolation-centric storage architecture that is proposed in Chapter 5, which leverages isolation mechanisms from different layers to facilitate secure storage for an enclave. Recommendations: The following highlights recommendations from the investigations done thus far. - The Lustre filesystem offers excellent performance but does not support some security related features, e.g., encryption, that are included in GPFS. If encryption is of paramount importance, then GPFS may be a more suitable choice. - There are several possible Lustre related enhancements that may provide functionality of use for secure-enclaves. However, since these features are not currently integrated, the use of Lustre as a secure storage system may require more direct involvement (support). (*The network that connects the storage subsystem and users, e.g., Lustre s LNET.) - The use of OpenStack with GPFS will be more streamlined than with Lustre, as there are available drivers for GPFS. - The Manilla project offers Filesystem as a Service for OpenStack and is worth further investigation. Manilla has some support for GPFS. - The proposed Lustre enhancement of Dynamic-LNET should be further investigated to provide more dynamic changes to the storage network which could be used to isolate hosts and their tenants. - The Linux namespaces offer a good solution for creating efficient restrictions to shared HPC filesystems. However, we still need to conduct a thorough round of storage/filesystem benchmarks. - Vendor products should be more closely reviewed, possibly to include evaluation of performance/protection of select products. (Note, we are investigation the option of evaluating equipment from Seagate/Xyratex.) Outline: The remainder of this report is structured as follows: - Section 1: Describes the growing importance of secure storage architectures and highlights some challenges for HPC. - Section 2: Provides background information on HPC storage architectures, relevant supporting technologies for secure storage and details on OpenStack components related to storage. Note, that background material on HPC storage architectures in this chapter can be skipped if the reader is already familiar with Lustre and GPFS. - Section 3: A review of protection mechanisms in two HPC filesystems; details about available isolation, authentication/authorization and performance capabilities are discussed. - Section 4: Describe technologies that can be used to bridge gaps in HPC storage and filesystems to facilitate...

Aderholdt, Ferrol↗
DOE OSTI2023

Modeling of a Networked Community Microgrid with High Solar Penetration Considering Control Dynamics of Inverter-Based Resources

Microgrids help facilitate the integration of renewable energy in distribution-level grids and increase the resilience of the electric grid to extreme weather, especially in rural areas. Compared to traditional microgrids, a networked microgrid leverages multiple grid-forming sources to form a potential meshed grid and is more flexible in operation. This paper demonstrates the simulation modeling of an actual networked microgrid located in Adjuntas, Puerto Rico. The model contains representations of power inverters that connect the battery energy storage systems and photovoltaic generation systems to the networked microgrid and is capable of simulating fast grid transients as well as long-term operation of the networked microgrid. The modeling technique for power inverters allows the time-efficient simulation of the microgrid with a minimal penalty on model accuracy.

Li, Dingrui↗
DOE OSTI2023

Design and implementation of I/O performance prediction scheme on HPC systems through large-scale log analysis

Abstract Large-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units used by hundreds to thousands of users simultaneously. Applications from large numbers of users have diverse characteristics, such as varying computation, communication, memory, and I/O intensity. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, I/O performance is becoming increasingly important as data sizes rapidly increase and large-scale applications, such as simulation and model training, are widely adopted. However, predicting I/O performance is difficult because I/O systems are shared among all users and involve many layers of software and hardware stack, including the application, network interconnect, operating system, file system, and storage devices. Furthermore, updates to these layers and changes in system management policy can significantly alter the I/O behavior of applications and the entire system. To improve the prediction of the I/O performance on HPC systems, we propose integrating information from several different system logs and developing a regression-based approach to predict the I/O performance. Our proposed scheme can dynamically select the most relevant features from the log entries using various feature selection algorithms and scoring functions, and can automatically select the regression algorithm with the best accuracy for the prediction task. The evaluation results show that our proposed scheme can predict the write performance with up to 90% prediction accuracy and the read performance with up to 99% prediction accuracy using the real logs from the Cori supercomputer system at NERSC.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Software Defined Cyberinfrastructure

Within and across thousands of science labs, researchers and students struggle to manage data produced in experiments, simulations, and analyses. Largely manual research data lifecycle management processes mean that much time is wasted, research results are often irreproducible, and data sharing and reuse remain rare. In response, we propose a new approach to data lifecycle management in which researchers are empowered to define the actions to be performed at individual storage systems when data are created or modified: actions such as analysis, transformation, copying, and publication. We term this approach software-defined cyberinfrastructure because users can implement powerful data management policies by deploying rules to local storage systems, much as software-defined networking allows users to configure networks by deploying rules to switches.We argue that this approach can enable a new class of responsive distributed storage infrastructure that will accelerate research innovation by allowing any researcher to associate data workflows with data sources, whether local or remote, for such purposes as data ingest, characterization, indexing, and sharing. We report on early experiments with this approach in the context of experimental science, in which a simple if-trigger-then-action (IFTA) notation is used to define rules.

Foster, Ian↗
DOE OSTI2024

OctoFAS: A Two-Level Fair Scheduler That Increases Fairness in Network-Based Key-Value Storage

We identified a fairness problem in a network-based key-value storage system using Intel Storage Performance Development Kit (SPDK) in a multitenant environment. In such an environment, each tenant’s I/O service rate is not fairly guaranteed compared to that of other tenants. To address the fairness problem, we propose OctoFAS, a two-level fair scheduler designed to improve overall throughput and fairness among tenants. The two-level scheduler of OctoFAS consists of (i) inter-core scheduling and (ii) intra-core scheduling. Through inter-core scheduling, OctoFAS addresses the load imbalance problem that is inherent in SPDK on the storage server by dynamically migrating I/O requests from overloaded cores to underloaded cores, thereby increasing overall throughput. Intra-core scheduling prioritizes handling requests from starving tenants over well-fed tenants within core-specific event queues to ensure fair I/O services among multiple tenants. OctoFAS is deployed on a Linux cluster with SPDK. Through extensive evaluations, we found that OctoFAS ensures that the total system throughput remains high and balanced, while enhancing fairness by approximately 10% compared to the baseline, when both scheduling levels operate in a hybrid fashion.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Nonlinear power flow control for networked AC/DC microgrids

A method for designing feedforward and feedback controllers for integration of stochastic sources and loads into a nonlinear networked AC/DC microgrid system is provided. A reduced order model for general networked AC/DC microgrid systems is suitable for HSSPFC control design. A simple feedforward steady state solution is utilized for the feedforward controls block. Feedback control laws are provided for the energy storage systems. A HSSPFC controller design is implemented that incorporates energy storage systems that provides static and dynamic stability conditions for both the DC random stochastic input side and the AC random stochastic load side. Transient performance was investigated for the feedforward/feedback control case. Numerical simulations were performed and provided power and energy storage profile requirements for the networked AC/DC microgrid system overall performance. The HSSPFC design can be implemented in the Matlab/Simulink environment that is compatible with real time simulation/controllers.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Framework for optimization of long-term, multi-period investment planning of integrated urban energy systems

In order to achieve stringent greenhouse gas emission reductions, a transition of our entire energy system from fossil to renewable resources needs to be designed. Such an energy transition brings two main challenges: most renewables generate variable electric energy, yet most demand is currently not electric (carrier mismatch) and does not always manifest at the same time as supply (temporal mismatch). Integrating multiple energy infrastructures can address both challenges by using the synergy between different energy carriers; building on existing infrastructure, while allowing a robust and flexible integration of the new. This paper proposes an optimization framework for long-term, multi-period investment planning of urban energy systems in an integrated manner. We formulate it as a mixed-integer linear program, combining a capacitated facility location with a multi-dimensional, capacitated network design problem. It includes generation and network expansion planning as well as interconnections between networks and storage infrastructure for each energy system. It can incorporate pathway effects like techno-economic developments, policy measures, and weather variations. The intended use is to support urban decision makers with long-term investment planning, though it can be tailored to fit other geographical or temporal scales. We demonstrate the model using two cases based on an average city in The Netherlands, which wants to reduce its CO 2 -emissions with 95% by 2050. In the first case, we include explicit carbon-emission constraints to study the effects of the carrier mismatch. In the second case, we implement interannual weather variations to analyze the temporal mismatch. The results give valuable insights into the energy transition design strategy for urban decision makers. They also show the future potential, as well as the computational challenges of the optimization framework.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2014

Flexibility in 21st Century Power Systems (Fact Sheet)

Flexibility of operation--the ability of a power system to respond to change in demand and supply--is a characteristic of all power systems. Flexibility is especially prized in twenty-first century power systems, with higher levels of grid-connected variable renewable energy (primarily, wind and solar). Sources of flexibility exist--and can be enhanced--across all of the physical and institutional elements of the power system, including system operations and markets, demand side resources and storage; generation; and transmission networks. Accessing flexibility requires significant planning to optimize investments and ensure that both short- and long-time power system requirements are met.

21ST CENTURY POWER SYSTEMS↗