Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “STORAGE AND NETWORKING SYSTEMS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRS1993

GE networked mass storage solutions supporting IEEE network mass storage model

The General Electric Government Communications Systems Department (GE/GCSD) has developed a near real time digital data storage and retrieval system that extends the capabilities currently available in today's marketplace. This system called DuraStore uses commercially available rotary tape drive technology with ANSI/IEEE standards for automated magnetic tape based data storage. It uses a nonproprietary approach to satisfy a wide range of data rates and storage capabilities requirements and is compliant with the IEEE Network Storage Model. Rotary tape drives, standard interfaces, application specific hardware/software, networked automated tape libraries, library administrator, write protection, volume/physical media linkages, and maximum resource utilization are addressed.

Herzog, Donald↗
DOE OSTI2022

Understanding the Impact of Data Staging for Coupled Scientific Workflows

We report the rate of data generated by cutting-edge experimental science facilities and large-scale simulations enabled by current high-performance computing (HPC) systems has continued to grow at a far greater pace than the development of the network and storage capabilities on which these systems rely. To cope with this challenge, scientist are moving toward the creation of autonomous experiments and HPC simulations using machine learning. However, efficiently moving, storing, and processing large amounts of data away from the point of origin presents an incredible challenge. In-memory computing, in situ analysis, data staging, and data streaming are recognized viable alternatives to traditional file-based methods for transferring data between coupled workflows. However, the performance trade-offs and limitations for these methods are not fully understood when used in HPC applications. This article presents a comprehensive performance assessment of the current solutions for data staging when applied to applications that are not necessary I/O intensive which makes them not ideal candidates for these methods. Our study is based on experiments running at scale on Oak Ridge National Laboratory's Summit supercomputer using applications and simulations that cover typical computational motifs and patterns. We investigated the usability and cost/benefit trade-offs of staging algorithms for HPC applications under different scenarios and highlight opportunities for optimizing the dataflow between coupled simulation workflows.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2015

High Resolution Nature Runs and the Big Data Challenge

NASA's Global Modeling and Assimilation Office at Goddard Space Flight Center is undertaking a series of very computationally intensive Nature Runs and a downscaled reanalysis. The nature runs use the GEOS-5 as an Atmospheric General Circulation Model (AGCM) while the reanalysis uses the GEOS-5 in Data Assimilation mode. This paper will present computational challenges from three runs, two of which are AGCM and one is downscaled reanalysis using the full DAS. The nature runs will be completed at two surface grid resolutions, 7 and 3 kilometers and 72 vertical levels. The 7 km run spanned 2 years (2005-2006) and produced 4 PB of data while the 3 km run will span one year and generate 4 BP of data. The downscaled reanalysis (MERRA-II Modern-Era Reanalysis for Research and Applications) will cover 15 years and generate 1 PB of data. Our efforts to address the big data challenges of climate science, we are moving toward a notion of Climate Analytics-as-a-Service (CAaaS), a specialization of the concept of business process-as-a-service that is an evolving extension of IaaS, PaaS, and SaaS enabled by cloud computing. In this presentation, we will describe two projects that demonstrate this shift. MERRA Analytic Services (MERRA/AS) is an example of cloud-enabled CAaaS. MERRA/AS enables MapReduce analytics over MERRA reanalysis data collection by bringing together the high-performance computing, scalable data management, and a domain-specific climate data services API. NASA's High-Performance Science Cloud (HPSC) is an example of the type of compute-storage fabric required to support CAaaS. The HPSC comprises a high speed Infinib and network, high performance file systems and object storage, and a virtual system environments specific for data intensive, science applications. These technologies are providing a new tier in the data and analytic services stack that helps connect earthbound, enterprise-level data and computational resources to new customers and new mobility-driven applications and modes of work. In our experience, CAaaS lowers the barriers and risk to organizational change, fosters innovation and experimentation, and provides the agility required to meet our customers' increasing and changing needs

big data analysis↗
NASA NTRS2000

Implementing Journaling in a Linux Shared Disk File System

In computer systems today, speed and responsiveness is often determined by network and storage subsystem performance. Faster, more scalable networking interfaces like Fibre Channel and Gigabit Ethernet provide the scaffolding from which higher performance computer systems implementations may be constructed, but new thinking is required about how machines interact with network-enabled storage devices. In this paper we describe how we implemented journaling in the Global File System (GFS), a shared-disk, cluster file system for Linux. Our previous three papers on GFS at the Mass Storage Symposium discussed our first three GFS implementations, their performance, and the lessons learned. Our fourth paper describes, appropriately enough, the evolution of GFS version 3 to version 4, which supports journaling and recovery from client failures. In addition, GFS scalability tests extending to 8 machines accessing 8 4-disk enclosures were conducted: these tests showed good scaling. We describe the GFS cluster infrastructure, which is necessary for proper recovery from machine and disk failures in a collection of machines sharing disks using GFS. Finally, we discuss the suitability of Linux for handling the big data requirements of supercomputing centers.

Preslan, Kenneth W.↗
DOE OSTI2023

Modeling of a Networked Community Microgrid with High Solar Penetration Considering Control Dynamics of Inverter-Based Resources

Microgrids help facilitate the integration of renewable energy in distribution-level grids and increase the resilience of the electric grid to extreme weather, especially in rural areas. Compared to traditional microgrids, a networked microgrid leverages multiple grid-forming sources to form a potential meshed grid and is more flexible in operation. This paper demonstrates the simulation modeling of an actual networked microgrid located in Adjuntas, Puerto Rico. The model contains representations of power inverters that connect the battery energy storage systems and photovoltaic generation systems to the networked microgrid and is capable of simulating fast grid transients as well as long-term operation of the networked microgrid. The modeling technique for power inverters allows the time-efficient simulation of the microgrid with a minimal penalty on model accuracy.

Li, Dingrui↗
DOE OSTI2023

Design and implementation of I/O performance prediction scheme on HPC systems through large-scale log analysis

Abstract Large-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units used by hundreds to thousands of users simultaneously. Applications from large numbers of users have diverse characteristics, such as varying computation, communication, memory, and I/O intensity. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, I/O performance is becoming increasingly important as data sizes rapidly increase and large-scale applications, such as simulation and model training, are widely adopted. However, predicting I/O performance is difficult because I/O systems are shared among all users and involve many layers of software and hardware stack, including the application, network interconnect, operating system, file system, and storage devices. Furthermore, updates to these layers and changes in system management policy can significantly alter the I/O behavior of applications and the entire system. To improve the prediction of the I/O performance on HPC systems, we propose integrating information from several different system logs and developing a regression-based approach to predict the I/O performance. Our proposed scheme can dynamically select the most relevant features from the log entries using various feature selection algorithms and scoring functions, and can automatically select the regression algorithm with the best accuracy for the prediction task. The evaluation results show that our proposed scheme can predict the write performance with up to 90% prediction accuracy and the read performance with up to 99% prediction accuracy using the real logs from the Cori supercomputer system at NERSC.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Software Defined Cyberinfrastructure

Within and across thousands of science labs, researchers and students struggle to manage data produced in experiments, simulations, and analyses. Largely manual research data lifecycle management processes mean that much time is wasted, research results are often irreproducible, and data sharing and reuse remain rare. In response, we propose a new approach to data lifecycle management in which researchers are empowered to define the actions to be performed at individual storage systems when data are created or modified: actions such as analysis, transformation, copying, and publication. We term this approach software-defined cyberinfrastructure because users can implement powerful data management policies by deploying rules to local storage systems, much as software-defined networking allows users to configure networks by deploying rules to switches.We argue that this approach can enable a new class of responsive distributed storage infrastructure that will accelerate research innovation by allowing any researcher to associate data workflows with data sources, whether local or remote, for such purposes as data ingest, characterization, indexing, and sharing. We report on early experiments with this approach in the context of experimental science, in which a simple if-trigger-then-action (IFTA) notation is used to define rules.

Foster, Ian↗
DOE OSTI2024

OctoFAS: A Two-Level Fair Scheduler That Increases Fairness in Network-Based Key-Value Storage

We identified a fairness problem in a network-based key-value storage system using Intel Storage Performance Development Kit (SPDK) in a multitenant environment. In such an environment, each tenant’s I/O service rate is not fairly guaranteed compared to that of other tenants. To address the fairness problem, we propose OctoFAS, a two-level fair scheduler designed to improve overall throughput and fairness among tenants. The two-level scheduler of OctoFAS consists of (i) inter-core scheduling and (ii) intra-core scheduling. Through inter-core scheduling, OctoFAS addresses the load imbalance problem that is inherent in SPDK on the storage server by dynamically migrating I/O requests from overloaded cores to underloaded cores, thereby increasing overall throughput. Intra-core scheduling prioritizes handling requests from starving tenants over well-fed tenants within core-specific event queues to ensure fair I/O services among multiple tenants. OctoFAS is deployed on a Linux cluster with SPDK. Through extensive evaluations, we found that OctoFAS ensures that the total system throughput remains high and balanced, while enhancing fairness by approximately 10% compared to the baseline, when both scheduling levels operate in a hybrid fashion.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1973

Study of hypervelocity meteoroid impact on orbital space stations

Structural damage resulting in hypervelocity impact of a meteorite on a spacecraft is discussed. Of particular interest is the backside spallation caused by such a collision. To treat this phenomenon two numerical schemes were developed in the course of this study to compute the elastic-plastic flow fracture of a solid. The numerical schemes are a five-point finite difference scheme and a four-node finite element scheme. The four-node finite element scheme proved to be less sensitive to the type of boundary conditions and loadings. Although further development work is needed to improve the program versatility (generalization of the network topology, secondary storage for large systems, improving of the coding to reduce the run time, etc.), the basic framework is provided for a utilitarian computer program which may be used in a wide variety of situations. Analytic results showing the program output are given for several test cases.

Leimbach, K. R.↗
NASA NTRS2017

Atmospheric Correction for Hyperspectral Ocean Color Retrieval with Application to the Hyperspectral Imager for the Coastal Ocean (HICO)

The classical multi-spectral Atmospheric Correction (AC) algorithm is inadequate for the new generation of spaceborne hyperspectral sensors such as NASA's first hyperspectral Ocean Color Instrument (OCI) onboard the anticipated Plankton, Aerosol, Cloud, ocean Ecosystem (PACE) satellite mission. The AC process must estimate and remove the atmospheric path radiance contribution due to the Rayleigh scattering by air molecules and scattering by aerosols from the measured top-of-atmosphere (TOA) radiance, compensate for the absorption by atmospheric gases, and correct for reflection and refraction of the air-sea interface. In this work, we present and evaluate an improved AC for hyperspectral sensors developed within NASA's Sea-viewing Wide Field-of-view Sensor (SeaWiFS) Data Analysis System software package (SeaDAS). The improvement is based on combining the classical AC approach of multi-spectral capabilities to correct for the atmospheric path radiance, extended to hyperspectral, with a gas correction algorithm to compensate for absorbing gases in the atmosphere, including water vapor. The SeaDAS-hyperspectral version is capable of operationally processing the AC of any hyperspectral airborne or spaceborne sensor. The new algorithm development was evaluated and assessed using the Hyperspectral Imager for Coastal Ocean (HICO) scenes collected at the Marine Optical BuoY (MOBY) site, and other SeaWiFS Bio-optical Archive and Storage System (SeaBASS) and AERosol Robotic NETwork - Ocean Color (AERONET-OC) coastal sites. A hyperspectral vicarious calibration was applied to HICO, showing the validity and consistency of HICO's ocean color products. The hyperspectral AC capability is currently available in SeaDAS to the scientific community at https://oceancolor.gsfc.nasa.gov/.

Ocean Color↗
DOE OSTI2020

Nonlinear power flow control for networked AC/DC microgrids

A method for designing feedforward and feedback controllers for integration of stochastic sources and loads into a nonlinear networked AC/DC microgrid system is provided. A reduced order model for general networked AC/DC microgrid systems is suitable for HSSPFC control design. A simple feedforward steady state solution is utilized for the feedforward controls block. Feedback control laws are provided for the energy storage systems. A HSSPFC controller design is implemented that incorporates energy storage systems that provides static and dynamic stability conditions for both the DC random stochastic input side and the AC random stochastic load side. Transient performance was investigated for the feedforward/feedback control case. Numerical simulations were performed and provided power and energy storage profile requirements for the networked AC/DC microgrid system overall performance. The HSSPFC design can be implemented in the Matlab/Simulink environment that is compatible with real time simulation/controllers.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Framework for optimization of long-term, multi-period investment planning of integrated urban energy systems

In order to achieve stringent greenhouse gas emission reductions, a transition of our entire energy system from fossil to renewable resources needs to be designed. Such an energy transition brings two main challenges: most renewables generate variable electric energy, yet most demand is currently not electric (carrier mismatch) and does not always manifest at the same time as supply (temporal mismatch). Integrating multiple energy infrastructures can address both challenges by using the synergy between different energy carriers; building on existing infrastructure, while allowing a robust and flexible integration of the new. This paper proposes an optimization framework for long-term, multi-period investment planning of urban energy systems in an integrated manner. We formulate it as a mixed-integer linear program, combining a capacitated facility location with a multi-dimensional, capacitated network design problem. It includes generation and network expansion planning as well as interconnections between networks and storage infrastructure for each energy system. It can incorporate pathway effects like techno-economic developments, policy measures, and weather variations. The intended use is to support urban decision makers with long-term investment planning, though it can be tailored to fit other geographical or temporal scales. We demonstrate the model using two cases based on an average city in The Netherlands, which wants to reduce its CO 2 -emissions with 95% by 2050. In the first case, we include explicit carbon-emission constraints to study the effects of the carrier mismatch. In the second case, we implement interannual weather variations to analyze the temporal mismatch. The results give valuable insights into the energy transition design strategy for urban decision makers. They also show the future potential, as well as the computational challenges of the optimization framework.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRSDate not supplied

Microservice Architecture for Cognitive Networks

This develops the concept of a cognitive network and describes a microservice based architecture which could be used to implement such a system. Delay tolerant networking (DTN) influences the design of the architecture as well as the networking scenarios that the system attempts to address. A cognitive storage and fragmentation service is developed based on existing artificial intelligence techniques such as Advantage Actor Critic (A2C) and Deep Q-Networks. The system is simulated using OpenAI Gym in a custom developed DTN environment.

cognitive networks↗
DOE OSTI2026

Field Verification of Grid-Forming Inverter-Based Power Plant Performance for Transmission System Reliability

This report describes some examples of the operation of two grid-forming (GFM) inverter-based photovoltaic-battery hybrid power plants connected to the electric system of the island of Kauai in Hawaii. Both plants are photovoltaic power plants with battery energy storage and are connected to the networked transmission system operated by Kauai Island Utility Cooperative (KIUC), along with several other power plants including an oil-fired plant, diesel generators, small hydroelectric plants, a biofuel plant, and several other photovoltaic plants.

14 SOLAR ENERGY↗
DOE OSTI2020

Ad Hoc File Systems for High-Performance Computing

Storage backends of parallel compute clusters are still based mostly on magnetic disks, while newer and faster storage technologies such as flash-based SSDs or non-volatile random access memory (NVRAM) are deployed within compute nodes. Including these new storage technologies into scientific workflows is unfortunately today a mostly manual task, and most scientists therefore do not take advantage of the faster storage media. One approach to systematically include nodelocal SSDs or NVRAMs into scientific workflows is to deploy ad hoc file systems over a set of compute nodes, which serve as temporary storage systems for single applications or longer-running campaigns. This paper presents results from the Dagstuhl Seminar 17202 “Challenges and Opportunities of User-Level File Systems for HPC” and discusses application scenarios as well as design strategies for ad hoc file systems using node-local storage media. The discussion includes open research questions, such as how to couple ad hoc file systems with the batch scheduling environment and how to schedule stage-in and stage-out processes of data between the storage backend and the ad hoc file systems. Also presented are strategies to build ad hoc file systems by using reusable components for networking and how to improve storage device compatibility. Various interfaces and semantics are presented, for example those used by the three ad hoc file systems BeeOND, GekkoFS, and BurstFS. Their presentation covers a range from file systems running in production to cutting-edge research focusing on reaching the performance limits of the underlying devices.

97 MATHEMATICS AND COMPUTING↗