Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Distribution Service”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 541 records · Page 30

Integrated System and Application Continuous Performance Monitoring and Analysis Capability

Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia's Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia's ElectroMagnetic Plasma In Realistic Environments ( EMPIRE ) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia's HPC users.

97 MATHEMATICS AND COMPUTING↗

Integrated System and Application Continuous Performance Monitoring and Analysis Capability (Final)

Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia’s Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia’s ElectroMagnetic Plasma In Realistic Environments (EMPIRE) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia’s HPC users.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Applicability of Digital Flight to the Operations of Self-Piloted Unmanned Aircraft Systems in the National Airspace System

Unmanned Aircraft Systems (UAS) hold great promise for a new era of specialized missions, including personal air transportation, cargo flight operations, aerial surveys, inspections, firefighting and more. The anticipated market growth is significant. To unlock its scalability and incumbent benefits requires a human to oversee multiple flights simultaneously, focusing on multi-vehicle mission management and relinquishing to autonomous systems their active role in controlling the aircrafts’ flight paths. Key to the realization of these scalability benefits is minimally-encumbered access to the National Airspace System (NAS), which poses some unique challenges for self-piloted UAS aircraft operations. These include the requirement for compatibility with existing airspace structures and operations including Visual Flight Rules (VFR) and Instrument Flight Rules (IFR), neither of which were developed to accommodate the unique needs and capabilities of UAS. This paper explores the applicability of Digital Flight to the operations of self-piloted UAS. As proposed by NASA, Digital Flight is a flight operations capability, enabled by a set of cooperative procedures and digital technologies, in which flight operators ensure flight-path safety through automated separation and flight path management in lieu of visual procedures and Air Traffic Control separation services. Flights operating under potentially-forthcoming rules of Digital Flight employ advanced automation technologies, information sharing, connectivity to operational data, and cooperative behaviors through distributed decision-making to maintain safety and achieve mission objectives. Designed for integration with VFR and IFR operations in shared NAS airspace, potentially as a third set of flight rules, Digital Flight may provide the mechanism for UAS operators – and all aircraft operators – to scale and diversify their operations beyond what is achievable under current regulations.

DFR↗

PoroTomo Natural Laboratory Horizontal and Vertical Distributed Acoustic Sensing Data

This dataset includes links to the PoroTomo DAS data in both SEG-Y and hdf5 (via h5py and HSDS with h5pyd) formats with tutorial notebooks for use. Data are hosted on Amazon Web Services (AWS) Simple Storage Service (S3) through the Open Energy Data Initiative (OEDI). Also included are links to the documentation for the dataset, Jupyter Notebook tutorials for working with the data as it is stored in AWS S3, and links to data viewers in OEDI for the horizontal (DASH) and vertical (DASV) DAS datasets. Horizontal DAS (DASH) data collection began 3/8/16, paused, and then started again on 3/11/2016 and ended 3/26/2016 using zigzag trenched fiber optic cabels. Vertical DAS (DASV) data collection began 3/17/2016 and ended 3/28/16 using a fiber optic cable through the first 363 m of a vertical well. These are raw data files from the DAS deployment at (DASH) and below (DASV) the surface during testing at the PoroTomo Natural Laboratory at Brady Hot Spring in Nevada. SEG-Y and hdf5 files are stored in 30 second files organized into directories by day. The hdf5 files available via HSDS are stored in daily files. Metadata includes information on the timing of recording gaps and a file count is included that lists the number of files from each day of recording. These data are available for download without login credentials through the free and publicly accessible Open Energy Data Initiative (OEDI) data viewer which allows users to browse and download individual or groups of files.

15 GEOTHERMAL ENERGY↗

Bayesian estimation of crack initiation times from service data

Lockheed C-130 Hercules aircraft have during their service life been periodically inspected and growing cracks around rivet holes were recorded. This record has recently been used to determine the statistical distributions of crack initiation times and the distribution of initial crack sizes. When crack initiation times are calculated from such cracks, by backward extrapolation of the growth relation, the resulting distribution of crack initiation times will indicate a preponderance of short times to crack initiation. If however, such distributions are combined with the reliability of the inspection procedure, the statistical distribution of missed initiation times can be estimated. The method used is based on Bayes theorem which permits the calculation of the 'prior' distribution (initiation times before inspection) from a knowledge of the 'posterior' distribution (initiation times obtained from the inspection) and a 'likelihood function' (reliability of the inspection) procedure. The results indicate that during an early inspection a large percentage of initiation times will be missed and that the fraction of located initiation times increases during later inspections.

Heller, R. A.↗

Autonomous Energy Grids

Current frameworks to monitor, control, and optimize large-scale energy systems are becoming increasingly inadequate because of significantly high penetration levels of variable generation and distributed energy resources being integrated into electric power systems; the deluge of data from pervasive metering of energy grids; and a variety of new market mechanisms, including multilevel ancillary services. This paper outlines the concept of autonomous energy grids (AEGs). These systems are supported by a scalable, reconfigurable, and self-organizing information and control infrastructure, are extremely secure and resilient (self-healing), and can self-optimize in real time to ensure economic and reliable performance while systematically integrating energy in all forms. AEGs rely on cellular building blocks that can self-optimize when isolated from a larger grid and participate in optimal operation when interconnected to a larger grid. This paper describes the key concepts and research necessary in the broad domains of optimization theory, control theory, big data analytics, and complex system theory and modeling to realize the AEG vision.

autonomous energy grids↗

MultiLoad-GAN: A GAN-Based Synthetic Load Group Generation Method Considering Spatial-Temporal Correlations

This paper presents a deep-learning framework, Multi-load Generative Adversarial Network (MultiLoad-GAN), for generating a group of synthetic load profiles (SLPs) simultaneously. The main contribution of MultiLoad-GAN is the capture of spatial-temporal correlations among a group of loads that are served by the same distribution transformer. This enables the generation of a large amount of correlated SLPs required for microgrid and distribution system studies. Here, the novelty and uniqueness of the MultiLoad-GAN framework are three-fold. First, to the best of our knowledge, this is the first method for generating a group of load profiles bearing realistic spatial- temporal correlations simultaneously. Second, two complementary realisticness metrics for evaluating generated load profiles are developed: computing statistics based on domain knowledge and comparing high-level features via a deep-learning classifier. Third, to tackle data scarcity, a novel iterative data augmentation mechanism is developed to generate training samples for enhancing the training of both the classifier and the MultiLoad-GAN model. Simulation results show that MultiLoad- GAN can generate more realistic load profiles than existing approaches, especially in group level characteristics. With little finetuning, MultiLoad-GAN can be readily extended to generate a group of load or PV profiles for a feeder or a service area.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Bike and Scoot to Transit

In this project, we developed a micromobility first-mile service for customers to access public transit. We launched this service as a pilot in the Seattle area that incentivized transit customers to bike or scoot to transit. The objective was to learn how to integrate different micromobility services and provide a unified reward program. Our pilot was called “Bike and Scoot to Transit” and ran from November 2022 to September 2023. More than a dozen locations were selected near transit hubs and light rail/train stations as preferred parking locations. Trips ending at those locations were partially funded. The pilot supported 19,226 qualified trips and distributed $73,000 in total benefits for the participants of the pilot. This dataset was collected from our pilot, which includes the following: - Monthly data: Monthly trip and funding summaries. - Data summary: Data broken down based on the micromobility service provider and equipment. - Trip data: List of all trips recorded during the pilot. - Pricing models: Fees charged by micromobility service providers. ![bike to transit](bike-transit.png)

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

The Role of National Research and Education Networks Providing Connectivity and Advanced Network Services to Virtual Communities in Collaborative R&E Projects. CUDI: The Mexican Case Part 1, , In: ISUM 2015: High Performance Computer Applications

This paper describes the role of National Research and Education Networks (NRENs) and their aim to provide a seamless, large scale, high-speed, low-latency network to accelerate scientific and education discovery for national, regional or global collaborative projects that are dependent on the use of modern high performance distributed computing systems (HPDCS) to process mathematical models, produce new knowledge and drive innovation when resolving complex problems. NRENs have become the mainstream of communication and collaboration to integrate the efforts of large geographically distributed teams of Virtual Communities (VC) that - train, teach and learn-, -create, use and share- knowledge, data, information and communications technology resources (e.g. computer and information systems, middleware, tools, instruments, data sets, repositories, applications and services) building e-infrastructures necessary to meet the collaborative projects’ goals. The paper focuses on the current status of the Mexican NREN: CUDI, its future organization and political support, necessary to provide seamless, large scale, high-speed, low latency and secure national backbone and international links providing connectivity, advanced network services and technical support to VCs and their e-infrastructures in collaborative R&E projects within the country, the LAC region and worldwide.

Computer Science↗

Designing and Implementing a Distributed System Architecture for the Mars Rover Mission Planning Software (Maestro)

Distributed systems allow scientists from around the world to plan missions concurrently, while being updated on the revisions of their colleagues in real time. However, permitting multiple clients to simultaneously modify a single data repository can quickly lead to data corruption or inconsistent states between users. Since our message broker, the Java Message Service, does not ensure that messages will be received in the order they were published, we must implement our own numbering scheme to guarantee that changes to mission plans are performed in the correct sequence. Furthermore, distributed architectures must ensure that as new users connect to the system, they synchronize with the database without missing any messages or falling into an inconsistent state. Robust systems must also guarantee that all clients will remain synchronized with the database even in the case of multiple client failure, which can occur at any time due to lost network connections or a user's own system instability. The final design for the distributed system behind the Mars rover mission planning software fulfills all of these requirements and upon completion will be deployed to MER at the end of 2005 as well as Phoenix (2007) and MSL (2009).

Goldgof, Gregory M.↗

Field To Farm Aggregation For Agricultural Systems

The Fields to Farms methodology illustrates the generation of farm parcels from the Crop Data Layer (CDL), a raster dataset containing 133 categories representing various crop types and land uses. This methodology involves two primary steps: Field Delineation and Farm Aggregation. The code specifically addresses the aggregation of pre-delineated fields within a county to form farms, adhering to predefined criteria for farm size categories. It is assumed that the field delineation process, which involves creating vector polygons from CDL raster, has been completed beforehand, possibly through external tools or methods. Upon initialization, the script processes county-level fields, preparing them for farm aggregation. In the Farm Aggregation phase, the code iteratively combines delineated fields into farms based on specified criteria, continuing until the aggregated farm size meets predefined thresholds derived from data from the 2017 National Agricultural Statistics Service (NASS) census. Throughout this iterative process, the script dynamically adjusts the aggregation to ensure alignment with the desired distribution reported by NASS. The resulting output of the script is a GeoDataFrame containing classified farms, which are subsequently saved as GeoPackage files. These files enable further analysis and visualization, facilitating comprehensive exploration of the farm landscape generated through the methodology.

Paudel, Rajiv [Idaho National Laboratory (INL), Id↗

eMosaic: Electrification Mosaic Platform for Grid Informed Smart Charging Management (Final Scientific/Technical Report)

ABB (Prime Contractor), in collaboration with its partners at the Utah State University (USU), Idaho National Laboratory, Rocky Mountain Power (RMP), and Electric Power Engineers (EPE), have performed research, development, and wide scale demonstration of a scalable and resilient Electrification Mosaic (eMosaic) platform for Smart Charge Management (SCM) for Electric Vehicle Infrastructure. Work was completed under DE EE0009194, titled “eMosaic Electrification Mosaic Platform for Grid Informed Smart Charging Management”, funded by the US Department of Energy. The project members developed algorithms that provide localized and bulk grid services and that reduce and stabilize costs all the way down the supply chain to the PEV owner through SCM. This platform aggregates telemetry from multiple data sources as pieces of the larger picture including personal, private fleet or transportation EVs, fast chargers and other supply equipment, weather service information, and geographically distributed charging sites such as public lots, garage and retail, and private or shared usage depots. ABB and the project team designed, tested, and improved a charging management system at local/edge and cloud levels. The ultimate objective of the project was to convincingly demonstrate that the developed secure eMosaic plat-form can be readily and favorably adopted by diverse utilities and site owners at scale. This was achieved through a demonstration plan with field deployment at several physical sites across 4 states and additional scalable simulation from high fidelity charging models.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Space Station Furnace Facility. Volume 2: Summary of technical reports

The Space Station Furnace Facility (SSFF) is a modular facility for materials research in the microgravity environment of the Space Station Freedom (SSF). The SSFF is designed for crystal growth and solidification research in the fields of electronic and photonic materials, metals and alloys, and glasses and ceramics, and will allow for experimental determination of the role of gravitational forces in the solidification process. The facility will provide a capability for basic scientific research and will evaluate the commercial viability of low-gravity processing of selected technologically important materials. In order to accommodate the furnace modules with the resources required to operate, SSFF developed a design that meets the needs of the wide range of furnaces that are planned for the SSFF. The system design is divided into subsystems which provide the functions of interfacing to the SSF services, conditioning and control for furnace module use, providing the controlled services to the furnace modules, and interfacing to and acquiring data from the furnace modules. The subsystems, described in detail, are as follows: Power Conditioning and Distribution Subsystem; Data Management Subsystem; Software; Gas Distribution Subsystem; Thermal Control Subsystem; and Mechanical Structures Subsystem.

Source record↗

R&D100: Lightweight Distributed Metric Service

On today's High Performance Computing platforms, the complexity of applications and configurations makes efficient use of resources difficult. The Lightweight Distributed Metric Service (LDMS) is monitoring software developed by Sandia National Laboratories to provide detailed metrics of system performance. LDMS provides collection, transport, and storage of data from extreme-scale systems at fidelities and timescales to provide understanding of application and system performance with no statistically significant impact on application performance.

97 MATHEMATICS AND COMPUTING↗

NASA's Fire Information for Resource Management System (FIRMS): Near Real-Time Global Fire Monitoring Using Data from MODIS and VIIRS

NASA's Fire Information for Resource Management System (FIRMS) provides near-real time active fire / hotspot products from MODIS and VIIRS to users in over 160 countries. The goal of FIRMS is to meet the needs of natural resource and protected area managers that face considerable challenges in obtaining timely satellite-derived information on fires burning within and around their management area. FIRMS has been reliably providing active fire / hotspot data in easy to use formats since its inception in 2006. Fire information is provided through a web map interface, email alerts, a web mapping service, and a range of downloadable files (SHP, CSV, KML and JSON). FIRMS data are used directly by end users and by brokers who take the data and add value to it before re-distributing it.FIRMS was initially developed by the University of Maryland in 2006; it was funded by the United Nations FAO and NASA's Applied sciences program under a NASA ROSES call. In 2012 FIRMS became integrated in to NASA's Land Atmosphere Near real-time Capability for EOS (LANCE); a virtual system that leverages NASAs existing science processing capabilities to deliver NRT data from ten instruments within 3 hours of satellite overpass. This presentation will describe the FIRMS system, provide an overview of the system, briefly describe some of the known applications and describe plans to further integrate the data in to NASA's Global Imagery Browse Services (GIBS) public mapping services and Worldview website.

active fire/hotspots↗

A Distributed Middleware Architecture for Attack-Resilient Communications in Smart Grids

Distributed Energy Resources (DERs) are being increasingly accepted as an excellent complement to traditional energy sources in smart grids. As most of these generators are geographically dispersed, dedicated communications investments for every generator are capital cost prohibitive. Real-time distributed communications middleware, which supervises, organizes and schedules tremendous amounts of data traffic in smart grids with high penetrations of DERs, allows for the use of existing network infrastructure. In this paper, we propose a distributed attack-resilient middleware architecture that detects and mitigates the congestion attacks by exploiting the Quality of Experience (QoE) measures to complement the conventional Quality of Service (QoS) information to detect and mitigate the congestion attacks effectively. The simulation results illustrate the efficiency of our proposed communications middleware architecture.

24 POWER TRANSMISSION AND DISTRIBUTION↗

NREL and San Diego Gas & Electric Company

San Diego Gas & Electric Company (SDG&EA(R)) and NREL are collaborating to evaluate advanced metering infrastructure (AMI)-based controls for grid operations. With millions of new smart meters being deployed around the country, this collaboration will take the lead in determining how to generate insights from the AMI data, and to use these insights for real-time operations. Smart meters ensure availability of pervasive and continuous data from the grid-edge, a service that is not possible with traditional grid sensors.

24 POWER TRANSMISSION AND DISTRIBUTION↗

HYDRA : High-speed simulation architecture for precision spacecraft formation simulation

e Hierarchical Distributed Reconfigurable Architecture- is a scalable simulation architecture that provides flexibility and ease-of-use which take advantage of modern computation and communication hardware. It also provides the ability to implement distributed - or workstation - based simulations and high-fidelity real-time simulation from a common core. Originally designed to serve as a research platform for examining fundamental challenges in formation flying simulation for future space missions, it is also finding use in other missions and applications, all of which can take advantage of the underlying Object-Oriented structure to easily produce distributed simulations. Hydra automates the process of connecting disparate simulation components (Hydra Clients) through a client server architecture that uses high-level descriptions of data associated with each client to find and forge desirable connections (Hydra Services) at run time. Services communicate through the use of Connectors, which abstract messaging to provide single-interface access to any desired communication protocol, such as from shared-memory message passing to TCP/IP to ACE and COBRA. Hydra shares many features with the HLA, although providing more flexibility in connectivity services and behavior overriding.

formation flying↗