Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data locality”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Improvement of aerosol optical depth data for localized solar resource assessment

Solar irradiance, especially for direct normal irradiance (DNI), is sensitive to the atmospheric aerosols, which can strongly extinguish DNI over areas with elevated aerosol loadings. The National Solar Radiation Database (NSRDB), developed by the National Renewable Energy Laboratory (NREL), uses AOD from MERRA-2 reanalysis and is further downscaled from 0.5° to 2-km based on elevation. However, the elevation-based downscaling may not accurately represent AOD, especially over the areas with large AOD gradients. This study examined whether the 1-km MODIS MAIAC satellite-retrieved AOD product can better represent the AOD distribution and be used to improve solar irradiance assessment. We focused on areas with relatively high AOD over North America (California, and New York City, US, and Mexico City in Mexico, in particular). The evaluation of MAIAC AOD and MERRA-2 AOD against ground-truth AERONET AOD shows that MAIAC AOD exhibits smaller RMSE by about 0.05 and higher correlation coefficient by 0.1-0.6 than MERRA-2 AOD. In addition, MERRA-2 AOD shows larger negative bias. The simulated DNI using MAIAC AOD (DNIMAIAC) and using MERRA-2 AOD (DNINSRDB) were evaluated with DNI observations. The results show the performance of DNIMAIAC is better than that of DNINSRDB with smaller RMSE by 0.5-1.5%, smaller positive mean bias by 0.8-3.1% and comparable correlation for the 3 sites analyzed. Overall, 1-km MAIAC AOD shows higher accuracy than 2-km elevation-based MERRA-2 AOD, leading to better performance of the simulated DNI using MAIAC AOD. Therefore, 1-km MAIAC AOD can be used to improve the accuracy of solar resource assessment.

14 SOLAR ENERGY↗
DOE OSTI2018

Scaling Structured Multigrid to 500K+ Cores Through Coarse-Grid Redistribution

The effcient solution of sparse, linear systems resulting from the discretization of partial differential equations is crucial to the performance of many physics-based simulations. The algorithmic optimality of multilevel approaches for common discretizations makes them a good candidate for an efficient parallel solver. Yet, modern architectures for high-performance computing systems continue to challenge the parallel scalability of multilevel solvers. While algebraic multigrid methods are robust for solving a variety of problems, the increasing importance of data locality and cost of data movement in modern architectures motivates the need to carefully exploit structure in the problem. Robust logically structured variational multigrid methods, such as Black Box Multigrid (BoxMG), maintain structure throughout the multigrid hierarchy. This avoids indirection and increased coarse- grid communication costs typical in parallel algebraic multigrid. Nevertheless, the parallel scalability of structured multigrid is challenged by coarse-grid problems where the overhead in communication dominates computation. In this paper, an algorithm is introduced for redistributing coarse-grid problems through incremental agglomeration. Guided by a predictive performance model, this algorithm provides robust redistribution decisions for structured multilevel solvers. A two-dimensional diffusion problem is used to demonstrate the significant gain in performance of this algorithm over the previous approach that used agglomeration to one processor. In addition, the parallel scalability of this approach

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Electricity Baseline 2022 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2022 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilized the appdirs Python dependency (https://pypi.org/project/appdirs/). This submission includes the background data used to generate the 2022 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: `python -c "import appdirs; print(appdirs.user_data_dir())"`). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2022 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; data inventory↗
DOE OSTI2025

Electricity Baseline 2021 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2021 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2021 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2021 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗
DOE OSTI2025

Electricity Baseline 2020 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2020 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2020 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2020 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗
NASA NTRS1984

Do the Arching HI Filaments Show in Local Reddening Data?

Probable dust counterparts of arching structures are investigated in two cases: (1) the Eridanus expanding shell; and (2) the extended Sco-Cen feature. Color excess data may be used to derive distance and dimension estimates of these structures. Color excesses from two areas SA 121 and SA 144 situated on the position of the 3.2 to 6.3 km/s velocity-slice are applied. Some stratification of the color excesses is noticed for both areas but extreme patchiness typifies particularly the dust in the SA 144 region.

Knude, J.↗
NASA NTRS2000

A neural network to retrieve the mesoscale instantaneous latent heat flux over oceans from SSM/I observations

The turbulent latent and sensible heat fluxes are necessary to study heat budget of the upper ocean or initialize ocean general circulation models. In order to retrieve the latent heat flux from satellite observations authors mostly use a bulk approximation of the flux whose parameters are derived from different instrument. In this paper, an approach based on artificial neural networks is proposed and compared to the bulk method on a global data set and 3 local data sets.

SSM/I latent heat flux air-sea interactions↗
NASA NTRS1999

A Parallel Compact Multi-Dimensional Numerical Algorithm with Aeroacoustics Applications

In this study we propose a novel method to parallelize high-order compact numerical algorithms for the solution of three-dimensional PDEs (Partial Differential Equations) in a space-time domain. For this numerical integration most of the computer time is spent in computation of spatial derivatives at each stage of the Runge-Kutta temporal update. The most efficient direct method to compute spatial derivatives on a serial computer is a version of Gaussian elimination for narrow linear banded systems known as the Thomas algorithm. In a straightforward pipelined implementation of the Thomas algorithm processors are idle due to the forward and backward recurrences of the Thomas algorithm. To utilize processors during this time, we propose to use them for either non-local data independent computations, solving lines in the next spatial direction, or local data-dependent computations by the Runge-Kutta method. To achieve this goal, control of processor communication and computations by a static schedule is adopted. Thus, our parallel code is driven by a communication and computation schedule instead of the usual "creative, programming" approach. The obtained parallelization speed-up of the novel algorithm is about twice as much as that for the standard pipelined algorithm and close to that for the explicit DRP algorithm.

Povitsky, Alex↗
DOE OSTI2022

Scalable access to shared files in a distributed system

Logic may pre-populate cache of compute nodes allocated to a user's session. Logic may access an image file of a global filesystem, the image file to comprise a first set of one or more files associated with a user session and a second set of one or more files upon which execution of the first set will depend. Logic may store the image file in local data storage of each compute node of a spanning tree of compute nodes allocated for the user session. Logic may unpack the image file based on the data. Logic may configure the local environment of each compute node in the spanning tree to identify a location in local memory of each of the files. And logic may initiate execution of a local filesystem service in each of the compute nodes of the spanning tree.

Oganezov, Alexander↗
NASA NTRS2010

ADAS Update and Maintainability

Since 2000, both the National Weather Service Melbourne (NWS MLB) and the Spaceflight Meteorology Group (SMG) have used a local data integration system (LOIS) as part of their forecast and warning operations. The original LOIS was developed by the Applied Meteorology Unit (AMU) in 1998 (Manobianco and Case 1998) and has undergone subsequent improvements. Each has benefited from three-dimensional (3-D) analyses that are delivered to forecasters every 15 minutes across the peninsula of Florida. The intent is to generate products that enhance short-range weather forecasts issued in support of NWS MLB and SMG operational requirements within East Central Florida. The current LDIS uses the Advanced Regional Prediction System (ARPS) Data Analysis System (AD AS) package as its core, which integrates a wide variety of national, regional, and local observational data sets. It assimilates all available real-time data within its domain and is run at a finer spatial and temporal resolution than current national or regional-scale analysis packages. As such, it provides local forecasters with a more comprehensive understanding of evolving fine-scale weather features. Over the years, the LDIS has become problematic to maintain since it depends on AMU-developed shell scripts that were written for an earlier version of the ADAS software. The goals of this task were to update the NWS MLB/SMG LDIS with the latest version of ADAS, incorporate new sources of observational data, and upgrade and modify the AMU-developed shell scripts written to govern the system. In addition, the previously developed ADAS graphical user interface (GUI) was updated. Operationally, these upgrades will result in more accurate depictions of the current local environment to help with short-range weather forecasting applications, while also offering an improved initialization for local versions of the Weather Research and Forecasting (WRF) model used by both groups.

Watson, Leela R.↗
DOE OSTI2016

Asynchronous in situ connected-components analysis for complex fluid flows

Fluid flow in porous media is at the heart of problems such as groundwater contamination and carbon sequestration, and presents an important challenge for scientific computing. For this class of problem, large three-dimensional simulations are performed to advance scientific inquiry. On massively parallel computing systems, the volume of data generated by such approaches can become a productivity bottleneck if the raw data generated from the simulation is analyzed in a post-processing step. We present a physics-based framework for in situ data reduction that is theoretically grounded in multiscale averaging theory. We show how task parallelism can be exploited to concurrently perform a variety of analysis tasks with data-dependent costs, including the generation of iso-surfaces, morphological analyses, and connected components analysis. A task management framework is constructed to launch asynchronous analysis threads, manage dependencies between different tasks, promote data locality and hide the impact of data transfers. The framework is applied to analyze GPU-based simulations of two-fluid flow in porous media, generating a set of averaged measures that represents the overall system behavior. We demonstrate how the approach can be applied to perform physically-consistent averaging over fluid subregions using connected components analysis. Simulations performed on Oak Ridge National Lab s Titan supercomputer are profiled to demonstrate the performance of the associated multi-threaded in situ analysis for typical production simulation of two-fluid flow.

Mcclure, James↗
DOE OSTI2016

Asynchronous In Situ Connected-Components Analysis for Complex Fluid flows, In: ISAV '16 Proceedings of the 2nd Workshop on In Situ Infrastructures for Enabling Extreme-scale Analysis and Visualization

The simulation of multiscale physics is an important challenge for scientific computing. For this class of problem, large three-dimensional simulations are performed to advance scientific inquiry. On massively parallel computing systems, the volume of data generated by such approaches can become a productivity bottleneck if the raw data generated from the simulation is analyzed in a post-processing step. To address this, we present a physics-based framework for in situ data reduction that is theoretically grounded in multiscale averaging theory. We show how task parallelism can be exploited to concurrently perform a variety of analysis tasks with data-dependent costs, including the generation of iso-surfaces, morphological analyses, and connected components analysis. All analyses are performed in parallel using distributed memory and use the same domain decomposition as the simulation. A task management framework is constructed to leverage available parallelism within a node for analysis. The capabilities of the framework are to launch asynchronous analysis threads, manage dependencies between different tasks, promote data locality and minimize the impact of data transfers. The framework is applied to analyze GPU-based simulations of two-fluid-phase flow in porous media, generating a set of averaged measures that represents the overall system behavior. We demonstrate how the approach can be applied to perform physically-consistent analysis over fluid sub-regions determined from connected components analysis. Simulations performed on Oak Ridge National Lab's Titan supercomputer are profiled to demonstrate the performance of the associated multi-threaded in situ analysis approach for typical production simulation of two-fluid-phase flow.

Computer Science↗
DOE OSTI2018

SharP Data Constructs: Data Constructs to Enable Data-Centric Computing

Extreme-scale applications (i.e., Big-Compute) are becoming increasingly data-intensive, i.e., producing and consuming increasingly large amounts of data. The HPC systems traditionally used for these applications are now used for Big-Data applications such as data analytics, social network analysis, machine learning, and genomics. As a consequence of these trends, the system architecture should be flexible and data-centric. This can already be witnessed in the pre-exascale systems with TBs of on-node hierarchical and heterogeneous memories, PBs of system memory, low-latency, high-throughput networks, and many threaded cores. As such, the pre-exascale systems suit the needs of both Big-Compute and Big-Data applications. Though the system architecture is flexible enough to support both Big-Compute and Big-Data, we argue there is a software gap. Particularly, we need data-centric abstractions to leverage the full potential of the system, i.e., there is a need for native support for data resilience, the ability to express data locality and affinity, mechanisms to reduce data movement, the ability to share data, and abstractions to express User's data usage and data access patterns. In this paper, we (i) show the need for taking a holistic approach towards data-centric abstractions, (ii) show how these approaches were realized in the SHARed data-structure centric Programming abstraction (SharP) library, a data-structure centric programming abstraction, and (iii) apply these approaches to a variety of applications that demonstrate its usefulness. Particularly, we apply these approaches to QMCPack and the Graph500 benchmark and demonstrate the advantages of this approach on extreme-scale systems.

Aderholdt, William Ferrol↗
NASA NTRSDate not supplied

Data Fusion for Urban Air Quality Assessment & Forecasting

This presentation provides an overview for our funded project with NASA's Health and Air Quality Applied Sciences Program. The project will expand an existing air quality data fusion tool implemented in Google Earth Engine (GEE) by our project team members at Sonoma Technology, Inc. (STI), a private air quality data company. We will expand the capabilities of this tool using new methods developed by the NASA GMAO which will give it the capability of providing sub-city scale resolution and hourly frequency estimates and forecasts of three key air quality indicators: surface-level particulate matter (PM2.5), nitrogen dioxide (NO2), and ozone (O3). We will combine a variety of Earth Observations including satellite data, global air quality forecasts, and local data from regulatory-grade monitors and/or low cost sensors. We will implement the new data fusion capabilities into the existing GEE tool in consultation with our end-users to best address their needs for sub-city scale air quality estimates and forecasts.

K Emma Knowland↗
NASA NTRSDate not supplied

Game Based Learning For Earth Science Applications Training

Current NASA Earth capacity development programs employ mechanisms ranging from online resource sharing, and virtual and in-person trainings to share knowledge. While these programs are highly successful at engaging individuals around the world – in 2018, over 8000 individuals and over 2000 institutions from all 50 US states and over 140 countries were engaged through over 150 projects and trainings – user feedback has highlighted the desire for expanded hands-on, practical experiences in incorporating NASA EO insights with localized data and actions. We aim to address this gap by leveraging the benefits of game-based learning to build user skills in integrating NASA and local EO data to guide decisions for climate resiliency and hazard planning. This project is being executed as a two-phase crowdsourced challenge: 1) Phase 1 will require a well-researched product concept that reflects an understanding of NASA’s Earth data and tools and user needs, and proposes an innovative and interactive game or extended reality experience to train users in identifying relevant NASA data and applying insights to their climate resiliency decisions; 2) Winners of Phase 1 will be provided seed funding to develop a working prototype of the product. We aim to award 1-3 final winners to support the development of more than one game, thereby ensuring that NASA's diverse audiences around the world can access training games that best suit their needs and capabilities. This EO training game project fits in the NASA Earth Science Applied Sciences Program’s Capacity Development Program, contributing to the program mission of “helping people around the world better understand [NASA’s Earth] data and find ways to use them” (https://appliedsciences.nasa.gov/what-we-do/capacity-building). The final training game will complement existing programmatic activities of workforce development, trainings, and collaborative projects, while providing the unique value of providing interactive experiences to users and collecting real-time data and feedback to improve NASA’s Earth applications’ products and services related to climate resilience.

Human centered design↗