Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data locality”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
NASA NTRS2011

The New Web-Based Hera Data Processing System at the HEASARC

The HEASARC at NASA/GSFC has provide an on-line astronomical data processing system called Hera for several years. Hera provides a complete data processing environment, including installed software packages, local data storage, and the CPU resources needed to process the user's data. The original design of Hera, however, has 2 requirements that has limited it's usefulness for some users, namely, that 1) the user must download and install a small helper program on their own computer before using Hera, and 2) Hera requires that several computer ports/sockets be allowed to communicate through any local firewalls on the users machine. Both of these restrictions can be problematic for some users, therefore we are now migrating Hera into a purely Web based environment which only requires a standard Web browser. The first release of Web Hera is now publicly available at http://heasarc.gsfc.nasa.gov/webheara/. It currently provides a standard graphical interface for running hundreds of different data processing programs that are available in the HEASARC's ftools software package. Over the next year we to add more features to Web Hera, including an interactive command line interface, and more display and line capabilities.

Pence, W.↗
DOE OSTI2016

On Fusing Recursive Traversals of K-d Trees

Loop fusion is a key program transformation for data locality optimization that is implemented in production compilers. But optimizing compilers currently cannot exploit fusion opportunities across a set of recursive tree traversal computations with producer-consumer relationships. In this paper, we develop a compile-time approach to dependence characterization and program transformation to enable fusion across recursively specified traversals over k-ary trees. We present the FuseT source-to-source code transformation framework to automatically generate fused composite recursive operators from an input program containing a sequence of primitive recursive operators. We use our framework to implement fused operators for MADNESS, Multiresolution Adaptive Numerical Environment for Scientific Simulation. We show that locality optimization through fusion can offer more than an order of magnitude performance improvement.

data locality↗
DOE OSTI2018

Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting

Hybrid cloud bursting (i.e., leasing temporary off-premise cloud resources to boost the overall capacity during peak utilization) can be a cost-effective way to deal with the increasing complexity of big data analytics, especially for iterative applications. However, the low throughput, high latency network link between the on-premise and off-premise resources (“weak link”) makes maintaining scalability difficult. While several data locality techniques have been designed for big data bursting on hybrid clouds, their effectiveness is difficult to estimate in advance. Yet such estimations are critical, because they help users decide whether the extra pay-as-you-go cost incurred by using the off-premise resources justifies the runtime speed-up. To this end, the current paper presents a performance model and methodology to estimate the runtime of iterative MapReduce applications in a hybrid cloud-bursting scenario. The paper focuses on the overhead incurred by the weak link at fine granularity, for both the map and the reduce phases. This approach enables high estimation accuracy, as demonstrated by extensive experiments at scale using a mix of real-world iterative MapReduce applications from standard big data benchmarking suites that cover a broad spectrum of data patterns. As a result, not only are the produced estimations accurate in absolute terms compared with experimental results, but they are also up to an order of magnitude more accurate than applying state-of-art estimation approaches originally designed for single-site MapReduce deployments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Affordability of Household Transportation Fuel Costs by Region and Socioeconomic Factors

Transportation energy is an important component of household budgets. Household vehicle fuel, one part of total transportation energy costs is over 3% of total household expenditures nationwide in the U.S. The average annual expenditure of over $2,000 for vehicle fuel (mostly gasoline) is comparable to the total average household expenditures for electricity and natural gas combined. However, these average values vary geographically, and lower income households can face higher energy cost burdens. The highly resolved data developed in this study reveals the variation of household energy burdens across the country and provides localized data to support local decision making.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS2010

Introducing Locality-Aware Computation into OpenMP

This paper presents our idea to introduce data locality feature into OpenMP. Given the facts that the memory systems are hierarchical while OpenMP is at, we believe that it is important to introduce new features to OpenMF to provide Open MP programmer capability to manage the data layout and align tasks and data as close as possible in modern architectures. We present the syntax and examples of the proposed features in this paper, and hope to enable further discussion of useful language features to keep OpenMP scalable in emerging architectures.

Huang, Lei↗
NASA NTRS2016

Architecture for Cognitive Networking within NASAs Future Space Communications Infrastructure

Future space mission concepts and designs pose many networking challenges for command, telemetry, and science data applications with diverse end-to-end data delivery needs. For future end-to-end architecture designs, a key challenge is meeting expected application quality of service requirements for multiple simultaneous mission data flows with options to use diverse onboard local data buses, commercial ground networks, and multiple satellite relay constellations in LEO, MEO, GEO, or even deep space relay links. Effectively utilizing a complex network topology requires orchestration and direction that spans the many discrete, individually addressable computer systems, which cause them to act in concert to achieve the overall network goals. The system must be intelligent enough to not only function under nominal conditions, but also adapt to unexpected situations, and reorganize or adapt to perform roles not originally intended for the system or explicitly programmed. This paper describes architecture features of cognitive networking within the future NASA space communications infrastructure, and interacting with the legacy systems and infrastructure in the meantime. The paper begins by discussing the need for increased automation, including inter-system collaboration. This discussion motivates the features of an architecture including cognitive networking for future missions and relays, interoperating with both existing endpoint-based networking models and emerging information-centric models. From this basis, we discuss progress on a proof-of-concept implementation of this architecture as a cognitive networking on-orbit application on the SCaN Testbed attached to the International Space Station.

space networks↗
NASA NTRS2016

Architecture for Cognitive Networking within NASA's Future Space Communications Infrastructure

Future space mission concepts and designs pose many networking challenges for command, telemetry, and science data applications with diverse end-to-end data delivery needs. For future end-to-end architecture designs, a key challenge is meeting expected application quality of service requirements for multiple simultaneous mission data flows with options to use diverse onboard local data buses, commercial ground networks, and multiple satellite relay constellations in LEO, GEO, MEO, or even deep space relay links. Effectively utilizing a complex network topology requires orchestration and direction that spans the many discrete, individually addressable computer systems, which cause them to act in concert to achieve the overall network goals. The system must be intelligent enough to not only function under nominal conditions, but also adapt to unexpected situations, and reorganize or adapt to perform roles not originally intended for the system or explicitly programmed. This paper describes an architecture enabling the development and deployment of cognitive networking capabilities into the envisioned future NASA space communications infrastructure. We begin by discussing the need for increased automation, including inter-system discovery and collaboration. This discussion frames the requirements for an architecture supporting cognitive networking for future missions and relays, including both existing endpoint-based networking models and emerging information-centric models. From this basis, we discuss progress on a proof-of-concept implementation of this architecture, and results of implementation and initial testing of a cognitive networking on-orbit application on the SCaN Testbed attached to the International Space Station.

space networks↗
DOE OSTI2025

FEDERATED LEARNING ON STOCHASTIC NEURAL NETWORKS

Federated learning is a machine learning paradigm that leverages edge computing on client devices to optimize models while maintaining user privacy by ensuring that local data remain on the device. However, since all data are collected by clients, federated learning is susceptible to latent noise in local datasets. Factors such as limited measurement capabilities or human errors may introduce inaccuracies in client data. To address this challenge, we propose the use of a stochastic neural network as the local model within the federated learning framework. Stochastic neural networks not only facilitate the estimation of the true underlying states of the data but also enable the quantification of latent noise. We refer to our federated learning approach, which incorporates stochastic neural networks as local models, as federated stochastic neural networks. In this work we will present numerical experiments demonstrating the performance and effectiveness of our method, particularly in handling nonindependent and identically distributed data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Federated Learning and Differential Privacy: What might AI-Enhanced co-design of microelectronics learn?

Data is a valuable commodity, and it is often dispersed over multiple entities. Sharing data or models created from the data is not simple due to concerns regarding security, privacy, ownership, and model inversion. This limitation in sharing can hinder model training and development. Federated learning can enable data or model sharing across multiple entities that control local data without having to share or exchange the data themselves. Differential privacy is a conceptual framework that brings strong mathematical guarantee for privacy protection and helps provide a quantifiable privacy guarantee to any data or models shared. The concepts of federated learning and differential privacy are introduced along with possible connections. Lastly, some open discussion topics on how federated learning and differential privacy can tied to AI-Enhanced co-design of microelectronics are highlighted.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS2018

Optimization of Elastodynamic Finite Integration Technique on Intel Xeon Phi Knights Landing Processors

This work describes the development and optimization of an implementation of an isotropic elastodynamic finite integration technique (EFIT) code for parallelized computation on Intel Knights Landing (KNL) hardware. EFIT is a numerical approach resulting in standard staggered-grid finite difference equations for the elastodynamic equations of motion to simulate bulk waves is solids. The computationally efficient simulation of elastodynamic wave propagation and interactions in aerospace materials is of high-interest in the fields of nondestructive evaluation (NDE) and structural health monitoring (SHM). Ultrasonic inspection uses an ultrasonic signal, generated at the surface of the material/structure via use of a piezoelectric transducer, to propagate sound waves into the material where it interacts with any existing defects, as well as with structural boundaries and any material inhomogeneity. Reflections from defects and boundaries are then measured by a transducer. Realistic ultrasound simulation tools can significantly aid the development and optimization of inspection techniques and can assist in the interpretation of experimental data. The optimization of an elastodynamics simulation code for the KNL Many Integrated Core processor was performed. The optimization focused on data locality and vectorization. Results show that tiling of the data to exploit the cache behavior and allow for significant utilization of the KNL hardware. The MPI implementation allows for a scalable implementation enabling large problems to be simulated. The model results were validated against theoretical dispersion curves to within 2% of the group velocity, and within 0.5% of the phase velocity of the A0 mode. Aggressive use of tiling, threading, and vectorization techniques allowed for dramatically improved time to solution.

William C. Schneck↗
DOE OSTI2023

Software-Hardware Co-design of Heterogeneous SmartNIC System for Recommendation Models Inference and Training

Deep Learning Recommendation Models (DLRMs) are critical applications in various domains and have evolved as one of the single largest machine learning applications. Trillions of DLRM parameters exceed the on-chip memory capacity of GPUs. Large-scale multi-node systems are required for distributed DLRM inference and training, which suffer from the all-to-all communication bottleneck, mainly limiting the scalability of ever-growing DLRMs. In recent years, SmartNICs have evolved with coupled computation and communication capabilities providing opportunities for a powerful heterogeneous device in the system. However, there isn't such a distributed system that fully leverages the abundant smartNIC resources that resolve the scalability issue of DLRMs. In this work, we proposed a software-hardware co-design of a heterogeneous smartNIC system that resolves the communication bottleneck of distributed DLRMs, mitigates the memory bandwidth pressure, and improves computation efficiency. We provide a set of smartNIC designs of cache systems (including local cache and remote cache) and smartNIC computation kernels which reduce data movement, relieve memory lookup intensity, and improve the GPU's computation efficiency. In addition, we propose a graph algorithm that improves the data locality of queries within batches which optimizes the overall system performance with higher data reuse. Our evaluation shows that our system achieves 2.1x latency speedup for inference and 1.6x throughput speedup for training.

Guo, Anqi↗
NASA NTRS2012

Global Swath and Gridded Data Tiling

This software generates cylindrically projected tiles of swath-based or gridded satellite data for the purpose of dynamically generating high-resolution global images covering various time periods, scaling ranges, and colors called "tiles." It reconstructs a global image given a set of tiles covering a particular time range, scaling values, and a color table. The program is configurable in terms of tile size, spatial resolution, format of input data, location of input data (local or distributed), number of processes run in parallel, and data conditioning.

Thompson, Charles K.↗
NASA NTRS1996

Thread Migration in the Presence of Pointers

Dynamic migration of lightweight threads supports both data locality and load balancing. However, migrating threads that contain pointers referencing data in both the stack and heap remains an open problem. In this paper we describe a technique by which threads with pointers referencing both stack and non-shared heap data can be migrated such that the pointers remain valid after migration. As a result, threads containing pointers can now be migrated between processors in a homogeneous distributed memory environment.

Cronk, David↗
DOE OSTI2021

Develop Wake Mitigation Strategy (CRADA CRD-17-00693 Final Report)

Wind turbines in a wind farm typically operate individually to maximize their own performance and do not take into account information from nearby turbines. In an autonomous wind farm, enabling cooperation to achieve farm-level objectives, turbines will need to use information from nearby turbines to optimize performance, ensure resiliency when other sensors fail, and adapt to changing local conditions. A key element of achieving an autonomous wind farm is to develop algorithms that provide necessary information to ensure reliable, robust, and efficient operation of wind turbines in a wind plant using local sensor information that is already being collected, such as supervisory control and data acquisition (SCADA) data, local meteorological stations, and nearby radars/sodars/lidars. In this work consensus control is applied in a hybrid analysis to data from an existing wind farm to demonstrate the benefit of consensus control.

17 WIND ENERGY↗
DOE OSTI2017

An electrostatic Particle-In-Cell code on multi-block structured meshes

We present an electrostatic Particle-In-Cell (PIC) code on multi-block, locally structured, curvilinear meshes called Curvilinear PIC (CPIC). Multi-block meshes are essential to capture complex geometries accurately and with good mesh quality, something that would not be possible with single-block structured meshes that are often used in PIC and for which CPIC was initially developed. In spite of the structured nature of the individual blocks, multi-block meshes resemble unstructured meshes in a global sense and introduce several new challenges, such as the presence of discontinuities in the mesh properties and coordinate orientation changes across adjacent blocks, and polyjunction points where an arbitrary number of blocks meet. In CPIC, these challenges have been met by an approach that features: (1) a curvilinear formulation of the PIC method: each mesh block is mapped from the physical space, where the mesh is curvilinear and arbitrarily distorted, to the logical space, where the mesh is uniform and Cartesian on the unit cube; (2) a mimetic discretization of Poisson's equation suitable for multi-block meshes; and (3) a hybrid (logical-space position/physical-space velocity), asynchronous particle mover that mitigates the performance degradation created by the necessity to track particles as they move across blocks. The numerical accuracy of CPIC was verified using two standard plasma–material interaction tests, which demonstrate good agreement with the corresponding analytic solutions. And compared to PIC codes on unstructured meshes, which have also been used for their flexibility in handling complex geometries but whose performance suffers from issues associated with data locality and indirect data access patterns, PIC codes on multi-block structured meshes may offer the best compromise for capturing complex geometries while also maintaining solution accuracy and computational efficiency.

36 MATERIALS SCIENCE↗
DOE OSTI2019

Localized Actual Meteorological Year File Creator (LAF): A tool for using locally observed weather data in building energy simulations

The Localized Actual Meteorological Year File Creator (LAF) application provides web-based access to real meteorological data and processes it into a weather file suitable for building energy modeling. Building energy consumption is affected by what is inside the building (such as occupants, appliances, HVAC systems, etc.) and by what is outside: the weather conditions the building is exposed to. However, freely available weather data files are limited to a number of specific locations, usually airports, which are often located away from city centers where buildings are concentrated. The authors have developed a new tool that supports the investigation and quantification of micro-climate conditions on building energy consumption. LAF is built on the Python open-source programming language and has a Graphical User Interface (GUI) that allows users to create custom weather data files for building energy simulations. Many sets of actual meteorological year weather data for long time periods are publicly available online (such as the MesoWest database) for thousands of locations in the US. LAF selects weather data according to user specifications and automatically processes it through an API across multiple weather stations and multiple time periods. The user may easily select a specific location, time frame, and time step that best meets their needs. This article presents a useful tool for energy modelers, building designers and operators to assist with building performance analysis and optimization.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2020

Extended experimental inferential structure determination method in determining the structural ensembles of disordered protein states

Proteins with intrinsic or unfolded state disorder comprise a new frontier in structural biology, requiring the characterization of diverse and dynamic structural ensembles. Here we introduce a comprehensive Bayesian framework, the Extended Experimental Inferential Structure Determination (X-EISD) method, which calculates the maximum log-likelihood of a disordered protein ensemble. X-EISD accounts for the uncertainties of a range of experimental data and back-calculation models from structures, including NMR chemical shifts, J-couplings, Nuclear Overhauser Effects (NOEs), paramagnetic relaxation enhancements (PREs), residual dipolar couplings (RDCs), hydrodynamic radii (R h ), single molecule fluorescence Förster resonance energy transfer (smFRET) and small angle X-ray scattering (SAXS). We apply X-EISD to the joint optimization against experimental data for the unfolded drkN SH3 domain and find that combining a local data type, such as chemical shifts or J-couplings, paired with long-ranged restraints such as NOEs, PREs or smFRET, yields structural ensembles in good agreement with all other data types if combined with representative IDP conformers.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2018

Evaluating Wind Direction Consensus Methods: A Case Study

Wind turbines in a wind farm typically operate individually to maximize their own performance and do not take into account information from nearby turbines. To enable cooperation to achieve farm-level objectives, turbines will need to use information from nearby turbines to optimize performance, ensure resiliency when other sensors fail, and adapt to changing local conditions. A key element of achieving these cooperative wind farms is to develop algorithms that provide necessary information to ensure reliable and efficient operation of turbines in a wind plant using local sensor information that is already being collected, such as supervisory control and data acquisition (SCADA) data, local meteorological stations, and nearby radars/sodars/lidars. This presentation demonstrates a framework for implementing a cooperative wind. Farm that incorporates information from local sensors in real time to better align turbines in a wind farm. Oftentimes, measurements made at an individual turbine are noisy and unreliable. By incorporating measurements from multiple nearby turbines, a more reliable estimate of the wind direction can be obtained at an individual turbine. Results indicate that this estimate of the wind direction can be used to improve the turbine's knowledge of the wind direction and could decrease dynamic yaw misalignment, decrease the amount of time a turbines spends yawing due to a more reliable input to the yaw controller, and increase resiliency to faulty wind-vane measurements. The concept was evaluated on data provided by WindESCo and was evaluated across several other averaging techniques to demonstrate the resiliency of the proposed consensus method.

17 WIND ENERGY↗