Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data storage data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

A Note on Interfacing Object Warehouses and Mass Storage Systems for Data Mining Applications

Data mining is the automatic discovery of patterns, associations, and anomalies in data sets. Data mining requires numerically and statistically intensive queries. Our assumption is that data mining requires a specialized data management infrastructure to support the aforementioned intensive queries, but because of the sizes of data involved, this infrastructure is layered over a hierarchical storage system. In this paper, we discuss the architecture of a system which is layered for modularity, but exploits specialized lightweight services to maintain efficiency. Rather than use a full functioned database for example, we use light weight object services specialized for data mining. We propose using information repositories between layers so that components on either side of the layer can access information in the repositories to assist in making decisions about data layout, the caching and migration of data, the scheduling of queries, and related matters.

Grossman, Robert L.↗

TAZeR: Hiding the Cost of Remote I/O in Distributed Scientific Workflows

A perennial bottleneck in distributed workflow analytics is long access latencies for remote data. We ask the question: assuming that data must be accessed remotely, can latencies be hidden? We present TAZeR, a framework that reduces data access latency while increasing data reuse. TAZeR transparently converts POSIX I/O into operations that interleave application work with data transfer, i.e. read, prefetching, and write stage-out. TAZeR ensures read data moves directly to application memory without synchronous intervention (soft zero-copy). TAZeR uses distributed bandwidth-aware staging to exploit reuse across application tasks and to manage the capacity constraints of fast hierarchical storage. We evaluate TAZeR on a High Energy Physics workflow that requests remote data at 48 Gb/s (over two 1 Gb/s WAN links) using complex access patterns. TAZeR is 12× and 22× faster than XRootD (state-of-the-art) and file copies (current approach), respectively; and within 7% of optimal. We discuss conditions when TAZeR can hide I/O accesses; and evaluate performance as effective staging sizes change.

Tazer, remote I/O, data staging↗

Optimization of tomographic reconstruction workflows on geographically distributed resources

New technological advancements in synchrotron light sources enable data acquisitions at unprecedented levels. This emergent trend affects not only the size of the generated data but also the need for larger computational resources. Although beamline scientists and users have access to local computational resources, these are typically limited and can result in extended execution times. Applications that are based on iterative processing as in tomographic reconstruction methods require high-performance compute clusters for timely analysis of data. Here, time-sensitive analysis and processing of Advanced Photon Source data on geographically distributed resources are focused on. Two main challenges are considered: (i) modeling of the performance of tomographic reconstruction workflows and (ii) transparent execution of these workflows on distributed resources. For the former, three main stages are considered: (i) data transfer between storage and computational resources, (i) wait/queue time of reconstruction jobs at compute resources, and (iii) computation of reconstruction tasks. These performance models allow evaluation and estimation of the execution time of any given iterative tomographic reconstruction workflow that runs on geographically distributed resources. For the latter challenge, a workflow management system is built, which can automate the execution of workflows and minimize the user interaction with the underlying infrastructure. The system utilizes Globus to perform secure and efficient data transfer operations. The proposed models and the workflow management system are evaluated by using three high-performance computing and two storage resources, all of which are geographically distributed. Workflows were created with different computational requirements using two compute-intensive tomographic reconstruction algorithms. Experimental evaluation shows that the proposed models and system can be used for selecting the optimum resources, which in turn can provide up to 3.13× speedup (on experimented resources). Furthermore, the error rates of the models range between 2.1 and 23.3% (considering workflow execution times), where the accuracy of the model estimations increases with higher computational demands in reconstruction tasks.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Evolution of a high-performance storage system based on magnetic tape instrumentation recorders

In order to provide transparent access to data in network computing environments, high performance storage systems are getting smarter as well as faster. Magnetic tape instrumentation recorders contain an increasing amount of intelligence in the form of software and firmware that manages the processes of capturing input signals and data, putting them on media and then reproducing or playing them back. Such intelligence makes them better recorders, ideally suited for applications requiring the high-speed capture and playback of large streams of signals or data. In order to make recorders better storage systems, intelligence is also being added to provide appropriate computer and network interfaces along with services that enable them to interoperate with host computers or network client and server entities. Thus, recorders are evolving into high-performance storage systems that become an integral part of a shared information system. Data tape has embarked on a program with the Caltech sponsored Concurrent Supercomputer Consortium to develop a smart mass storage system. Working within the framework of the emerging IEEE Mass Storage System Reference Model, a high-performance storage system that works with the STX File Server to provide storage services for the Intel Touchstone Delta Supercomputer is being built. Our objective is to provide the required high storage capacity and transfer rate to support grand challenge applications, such as global climate modeling.

Peters, Bruce↗

Fourth NASA Goddard Conference on Mass Storage Systems and Technologies

This report contains copies of all those technical papers received in time for publication just prior to the Fourth Goddard Conference on Mass Storage and Technologies, held March 28-30, 1995, at the University of Maryland, University College Conference Center, in College Park, Maryland. This series of conferences continues to serve as a unique medium for the exchange of information on topics relating to the ingestion and management of substantial amounts of data and the attendant problems involved. This year's discussion topics include new storage technology, stability of recorded media, performance studies, storage system solutions, the National Information infrastructure (Infobahn), the future for storage technology, and lessons learned from various projects. There also will be an update on the IEEE Mass Storage System Reference Model Version 5, on which the final vote was taken in July 1994.

Kobler, Benjamin↗

A workflow to assess the efficacy of brine extraction for managing injection-induced seismicity potential using data from a CO 2 injection site near Decatur, Illinois

Injection of CO 2 for storage in a geologic formation increases pore pressure and alters in situ stresses. Depending on the orientation of any existing fault and fracture planes, such as critically stressed planes, this stress alteration will modify normal stresses acting on planes and could result in frictional sliding and release stored energy in the form of seismicity. Brine extraction (BE) is a technique that can be applied prior to, or during, CO 2 injection to reduce pore pressure for increasing storage capacity and, potentially, for reducing the likelihood of frictional sliding. Here a workflow is described to assess the efficacy of BE for mitigating frictional sliding (i.e., seismicity) during injection and entails: site characterization, stress calculations and failure assessment, static and dynamic modeling, and BE operational planning. Site characterization describes the stress field used to calculate the Coulomb Failure Function (CFF) that constrains allowable pore pressure changes and injection rates in the numerical simulation of CO 2 injection scenarios. The inclusion of BE in the workflow allows for determination of the potential need for pressure reduction, and evaluation of the effectiveness of this operation. Example application of the workflow using an injection field dataset near Decatur, IL, provides insight on fracture planes and stresses at the site, formation properties and the impact of variable CO 2 injection-rate targets on whether BE plans are required. The study workflow indicates that BE could enhance CO 2 injection rate by 39% and correspondingly reduce the potential for injection-induced seismicity as indicated by a reduction in CFF.

58 GEOSCIENCES↗

Intelligent Monitoring System for Real-time Geologic Storage, Optimization, and Reservoir Management

The objective of the subtask was to develop a near-real-time monitoring system for seismic data at the Decatur, IL, geologic carbon sequestration (GCS) site and specifically include fiber-optic cable derived distributed acoustic signal (DAS) data in the process. Owing to the large volumes of data, we opted to utilize existing deep borehole conventional seismic sensors for detection and pull DAS and shallow borehole seismic data once a detection has been made. Unfortunately, the horizontal fiber-optic cables did not yield microseismic signals for use in locating events near the GCS site. Various stacking and filtering approaches were tested without any coherent detection becoming apparent. We attribute the insensitivity to local microseismic events to a lack of coupling in downhole, vertical cable and the non-ideal alignment of the horizontal fiber-optic cable to the vertically polarized seismic energy. Despite the inability to detect and utilize the DAS data from the fiber-optic cables, we developed a general processing framework that enables easy adaptation for future deployment of fiber-optic cables with better suited alignment at Decatur or elsewhere.

Kaven, J. Ole↗

Quantifying Water Storage Change and Land Subsidence Induced by Reservoir Impoundment Using GRACE, Landsat, and GPS Data

The construction of hydropower dams is a common strategy to support a country's increasing need for electricity and river water management for industry and agriculture. Although the hydrological and geophysical impacts of water relocation are usually assessed prior to impoundment, their accuracy is generally limited due to the lack of in situ observations, especially in a remote area. This study presents a workflow to quantify the terrestrial water storage change (TWS) and land subsidence induced by a reservoir's water impoundment using multiple satellite observations (GRACE, Landsat), land surface models (CABLE, GLDAS, NCEP, ECMWF), and GPS data. The study site is the Bakun Dam, located in Sarawak, Malaysia, which is the largest hydropower dam in Southeast Asia. Commencing operation in late 2010, the dam induced a change of water mass and lake surface area that was clearly observed by GRACE and Landsat observations, respectively. During the 17-month impounding period (from August 2010 to December 2011), GRACE observed a dramatic increase of approximately 200 mm equivalent water height, while Landsat detected an increased lake extent of around 600 km2. In this paper, a forward model is developed to determine the increased water surface level corresponding to GRACE observations, estimated to be about 120 m. In contrast to GRACE, the TWS derived from land surface models cannot capture the increased TWS, due to the lack of reservoir routing algorithms in the models. In addition, the land subsidence was calculated using the disk load model constructed based on the GRACE-derived lake level and Landsat-derived lake extent; the result is validated with the GPS data from BIN1 station, located at the western coast of Borneo. The commencement stage of the Bakun Dam induces the large-scale land subsidence, which causes the GPS-BIN1 station to subside by ~9 mm, and move toward the Bakun Lake by ~4 mm. Computation of the surface displacements directly from GRACE spherical harmonic coefficient data fails to capture the subsidence feature, mainly due to the truncation error. Overall, this study demonstrates that evaluating GRACE in conjunction with Landsat, LSMs, and GPS data allows the exploitation of the gravity signal at a much smaller spatial scale than its intrinsic resolution. Benefiting from global coverage, the newly developed satellite-based algorithm is a valuable tool for assessing the impacts of reservoir operation on hydrological and geophysical changes from local to regional scales.

Natthachet Tangdamrongsub↗

Kepler Science Operations Center Pipeline Framework

The Kepler mission is designed to continuously monitor up to 170,000 stars at a 30 minute cadence for 3.5 years searching for Earth-size planets. The data are processed at the Science Operations Center (SOC) at NASA Ames Research Center. Because of the large volume of data and the memory and CPU-intensive nature of the analysis, significant computing hardware is required. We have developed generic pipeline framework software that is used to distribute and synchronize the processing across a cluster of CPUs and to manage the resulting products. The framework is written in Java and is therefore platform-independent, and scales from a single, standalone workstation (for development and research on small data sets) to a full cluster of homogeneous or heterogeneous hardware with minimal configuration changes. A plug-in architecture provides customized control of the unit of work without the need to modify the framework itself. Distributed transaction services provide for atomic storage of pipeline products for a unit of work across a relational database and the custom Kepler DB. Generic parameter management and data accountability services are provided to record the parameter values, software versions, and other meta-data used for each pipeline execution. A graphical console allows for the configuration, execution, and monitoring of pipelines. An alert and metrics subsystem is used to monitor the health and performance of the pipeline. The framework was developed for the Kepler project based on Kepler requirements, but the framework itself is generic and could be used for a variety of applications where these features are needed.

Klaus, Todd C.↗

Photovoltaic Analysis and Response Support (PARS) Platform for Solar Situational Awareness and Resiliency Services

The project's primary objective is to develop a digital-twin based Photovoltaic (PV) Analysis and Response Support (PARS) platform, which aims to provide real-time situational awareness and optimal response plans. This platform is designed to enhance the performance of hybrid PV systems, making them competitive with or even superior to conventional generation resources. The PARS platform enabled the project team to develop and evaluate an extensive suite of grid support functionalities for the hybrid PV systems to enhance grid performance, across key areas including visibility, dispatchability, security, resilience, and reliability. Given the global push toward achieving 100% clean energy by 2035, there is a significant increase in the integration of inverter-based resources (IBRs) throughout the energy grid. Effectively managing the inherent variability and uncertainty associated with IBRs is crucial for ensuring cost-effectiveness, reliability, and security in both the main grid and islanded microgrids. Constrained to a limited array of IEEE test systems or standard feeder models, traditional IBR modeling struggles to assimilate new field data, accurately reflect system dynamics, and adapt to the evolving energy landscape. In our project, we embraced a Digital Twin (DT) strategy for crafting the PARS platform. A digital twin acts as a precise virtual counterpart of a physical system, built on historical data and continuously honed with real-time insights. This enables the high-fidelity DT to accurately mirror current system operations and forecast future scenarios. Consequently, the PARS platform becomes an ideal environment for testing and refining monitoring, control, power, and energy management algorithms designed to boost hybrid PV system performance. The defining feature of the PARS platform, distinguishing it from other advanced simulation tools, is its exceptional adaptability. This is achieved by employing actual network topologies and utilizing real-time field data for fine-tuning and calibration, ensuring a close emulation of real-world conditions. The project deliverables include: 1) High-fidelity IBR models and tools for real-time parameterization, utilizing real-time field measurements to refine IBR models for enhanced accuracy and performance; 2) Grid-forming and Grid-following capabilities to deliver resilience services, including blackstart, voltage and frequency support, cold-load pick-up, power reserves, and three-phase load balancing across grid-connected and microgrid settings; 3) Machine learning-based forecasting tools and methods for generating synthetic data and topologies, creating diverse and realistic simulation environments for evaluating varied operational scenarios; 4) Advanced microgrid power and energy management algorithms for optimizing the integration and operation of PV, storage, and demand response resources within both feeder and community scales. The power grid data sets are provided by four utility companies in North Carolina and the New York Power Administration. Acting as industry advisors, our industry partners communicated stakeholder needs and regulatory standards to the research teams, aiding technology transfer by incorporating the developed methodologies into their daily operations. This collaboration ensures that the PARS platform, functioning as a power system digital twin, enhances our understanding of IBR dynamic behaviors and enables the development and evaluation of IBR control functions that match or exceed the capabilities of conventional synchronous generators.

14 SOLAR ENERGY↗

2023 Real-time Optimization Workflow Status Update

Nuclear integrated energy systems are composed of a diverse set of energy generation sources and exist in dynamic and competitive electricity markets. With the inclusion of thermal energy storage, nuclear power systems can store heat for future use through various processes, such as water desalination or hydrogen generation. This heat storage can be managed in such a way as to economically optimize its usage. By combining real-time price data from the day-ahead and real-time markets with predictive and intelligent models, the charging and discharging of the thermal energy storage may be determined and optimized. This research details an approach through models and systems for the real-time optimization (RTO) of capacity allocation. Virtual models of the energy system and its physics phenomenon and component interactions provide intelligence to verification and prediction of operations. An optimization framework can use data generated from both a set of physical assets as well as the virtual models to predict future performance and create optimization and control workflows. Data warehouse technologies can be used to combine data across models, optimization workflows, market price data, and sensor data to intuitively store various types of data and provide integrations to physical control systems as well as user visualizations. Put together, these components can create a system for the RTO of nuclear integrated energy systems. Various virtual models and bench-scale physical demonstrations have been successfully performed and verified using this system. Larger scale testbeds that include thermal energy storage systems have been identified as future opportunities. A gap analysis that details the steps necessary to reach a physical demonstration at this scale is provided, along with conclusions on the current effort and future work.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

HPSS in the Extreme Scale Era: Report to DOE Office of Science on HPSS in 2018-2022

This paper is a product for the Department of Energy?s (DOE) Office of Science (OS) reporting on the feasibility of using HPSS into the Extreme Scale era of storage (2018 -2022). The initial sections provide a summary of the systems environment and expected archival storage requirements extracted from other Extreme Scale workshopreports conducted since 2007 by various applications and programs within the DOE OS. These high level requirements aid in identifying long-term data storage system features that support Extreme Scale science. Participants also separately forecasted data growth in established long-term data storage systems through 2018 - 2022 to get a picture of the amount of data that systems will need to manage. The report concludes that HPSS is well positioned to meet the requirements projected for the Extreme Scale era and provides recommendations from the HPSS Collaboration to the DOE Office of Science for ensuring that HPSS can meet these extreme scale storage requirements of 2018 - 2022.

97 MATHEMATICS AND COMPUTING↗

NASA Remote Sensing Data in Earth Sciences: Processing, Archiving, Distribution, Applications at the GES DISC

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is one of the major Distributed Active Archive Centers (DAACs) archiving and distributing remote sensing data from the NASA's Earth Observing System. In addition to providing just data, the GES DISC/DAAC has developed various value-adding processing services. A particularly useful service is data processing a t the DISC (i.e., close to the input data) with the users' algorithms. This can take a number of different forms: as a configuration-managed algorithm within the main processing stream; as a stand-alone program next to the on-line data storage; as build-it-yourself code within the Near-Archive Data Mining (NADM) system; or as an on-the-fly analysis with simple algorithms embedded into the web-based tools (to avoid downloading unnecessary all the data). The existing data management infrastructure at the GES DISC supports a wide spectrum of options: from data subsetting data spatially and/or by parameter to sophisticated on-line analysis tools, producing economies of scale and rapid time-to-deploy. Shifting processing and data management burden from users to the GES DISC, allows scientists to concentrate on science, while the GES DISC handles the data management and data processing at a lower cost. Several examples of successful partnerships with scientists in the area of data processing and mining are presented.

Leptoukh, Gregory G.↗

Options for improving the load matching capability of distributed photovoltaics: Methodology and application to high-latitude data

At high latitudes, domestic electricity demand and insolation are negatively correlated on both an annual and a diurnal basis. With increasing integration of distributed photovoltaics (PV) in low-voltage distribution grids of residential areas, limits to the penetration level are set by voltage rise due to unmatched production and load. In this paper a methodology for determining the impacts of three options for increased load matching is presented and applied to high-latitude data. The studied options are PV array orientation, demand side management (DSM) and electricity storage. Detailed models for domestic electricity demand and PV output are used. An optimisation approach is applied to find an optimal distribution of PV systems on different array orientations and a best-case evaluation of DSM and a storage model are implemented. At high penetration levels, storage is the most efficient option for maximising the solar fraction, but at lower overproduction levels, the impact of DSM is equal or slightly better. An east-west orientation of PV arrays is suggested for high penetration levels, but the effect of the optimised orientation is small. Without an optimised storage operation, the overproduced power is more efficiently reduced by DSM than storage, although this is highly dependent on the applied DSM algorithm. Further research should be focused on the DSM potential and optimal operation of storage. (author)

14 SOLAR ENERGY↗

OSiRIS: a distributed Ceph deployment using software defined networking for multi-institutional research

We report on the first year of the OSiRIS project (NSF Award #1541335, UM, IU, MSU and WSU) which is targeting the creation of a distributed Ceph storage infrastructure coupled together with software-defined networking to provide high-performance access for wellconnected locations on any participating campus. The projects goal is to provide a single scalable, distributed storage infrastructure that allows researchers at each campus to read, write, manage and share data directly from their own computing locations. The NSF CC*DNI DIBBS program which funded OSiRIS is seeking solutions to the challenges of multi-institutional collaborations involving large amounts of data and we are exploring the creative use of Ceph and networking to address those challenges. While OSiRIS will eventually be serving a broad range of science domains, its first adopter will be the LHC ATLAS detector project via the ATLAS Great Lakes Tier-2 (AGLT2) jointly located at the University of Michigan and Michigan State University. Part of our presentation will cover how ATLAS is using the OSiRIS infrastructure and our experiences integrating our first user community. The presentation will also review the motivations for and goals of the project, the technical details of the OSiRIS infrastructure, the challenges in providing such an infrastructure, and the technical choices made to address those challenges. We will conclude with our plans for the remaining 4 years of the project and our vision for what we hope to deliver by the projects end.

97 MATHEMATICS AND COMPUTING↗

Grid Data Crossing Platform (GriD-Xing)

This project is to develop a data management and analytics platform called Grid Data Crossing (GriD-Xing) that integrates data processing, storage, analysis, and visualization functionalities for users at industry (e.g., EMS and DMS) and universities. It intends to increase the applicability of large set of multi-source data (e.g., SCADA, PMU, μPMU, AMI, inverter, load/DER forecasting, and cyber data) and eventually improve the resilience, reliability, and security of critical energy infrastructure. GriD-Xing also intends to be a research platform to identify the usefulness of large multi-source data via case studies of interest to industry (e.g., 2011 southwest cold weather and 2016 Blue Cut fire events). In the first year, the team focused on the development and demonstration of large multi-source data (e.g., PMU, μPMU, SCADA, AMI, and cyber data) processing, storage, and analysis. Specifically, the team designed GriD-Xing framework in functional and structural aspects, developed GriD-Xing algorithms for large multi-source grid data processing, storage, and analysis, and finally demonstrated GriD-Xing prototype algorithms performance through a series of tests and case studies using real-world and simulation data.

24 POWER TRANSMISSION AND DISTRIBUTION↗