Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data locality”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Enhancing Network Anomaly Detection Using Graph Neural Networks

In the world of Internet of Things (IoT) networks, where devices are constantly communicating, keeping them secure from cyber threats is critical. This paper introduces a novel approach to detecting unusual and potentially harmful activities in these networks using graph neural networks (GNNs). We combine two specific types of GNNs-GraphSAGE and graph attention networks (GAT)-to create a model that understands and represents the behaviors and interactions in a network. GraphSAGE creates an embedding of network activities by examining local data interactions, while GAT directs the model's focus to the most critical interactions. By integrating these two methods in a single model that considers different types of interactions (both host and flow nodes), we aim to create a system that accurately represents the current state of a network and can also spot anomalies effectively while reducing false positives and negatives. Our innovative approach has demonstrated promising results, achieving an accuracy of 98% on the UNSW-NB15 dataset, significantly outperforming standalone GraphSAGE and GAT models. This underscores its potential as a robust framework for securing IoT networks against cyber threats and anomalies.

Marfo, William↗

h5bench: A unified benchmark suite for evaluating HDF5 I/O performance on pre‐exascale platforms

Summary Parallel I/O is a critical technique for moving data between compute and storage subsystems of supercomputers. With massive amounts of data produced or consumed by compute nodes, high‐performant parallel I/O is essential. I/O benchmarks play an important role in this process; however, there is a scarcity of I/O benchmarks representative of current workloads on HPC systems. Toward creating representative I/O kernels from real‐world applications, we have created h5bench , a set of I/O kernels that exercise hierarchical data format version 5 (HDF5) I/O on parallel file systems in numerous dimensions. Our focus on HDF5 is due to the parallel I/O library's heavy usage in various scientific applications running on supercomputing systems. The various tests benchmarked in the h5bench suite include I/O operations (read and write), data locality (arrays of basic data types and arrays of structures), array dimensionality (one‐dimensional arrays, two‐dimensional meshes, three‐dimensional cubes), I/O modes (synchronous and asynchronous). In this paper, we present the observed performance of h5bench executed along several of these dimensions on existing supercomputers (Cori and Summit) and pre‐exascale platforms (Perlmutter, Theta, and Polaris). h5bench measurements can be used to identify performance bottlenecks and their root causes and evaluate I/O optimizations. As the I/O patterns of h5bench are diverse and capture the I/O behaviors of various HPC applications, this study will be helpful to the broader supercomputing and I/O community.

97 MATHEMATICS AND COMPUTING↗

Improvement of aerosol optical depth data for localized solar resource assessment

Solar irradiance, especially for direct normal irradiance (DNI), is sensitive to the atmospheric aerosols, which can strongly extinguish DNI over areas with elevated aerosol loadings. The National Solar Radiation Database (NSRDB), developed by the National Renewable Energy Laboratory (NREL), uses AOD from MERRA-2 reanalysis and is further downscaled from 0.5° to 2-km based on elevation. However, the elevation-based downscaling may not accurately represent AOD, especially over the areas with large AOD gradients. This study examined whether the 1-km MODIS MAIAC satellite-retrieved AOD product can better represent the AOD distribution and be used to improve solar irradiance assessment. We focused on areas with relatively high AOD over North America (California, and New York City, US, and Mexico City in Mexico, in particular). The evaluation of MAIAC AOD and MERRA-2 AOD against ground-truth AERONET AOD shows that MAIAC AOD exhibits smaller RMSE by about 0.05 and higher correlation coefficient by 0.1-0.6 than MERRA-2 AOD. In addition, MERRA-2 AOD shows larger negative bias. The simulated DNI using MAIAC AOD (DNIMAIAC) and using MERRA-2 AOD (DNINSRDB) were evaluated with DNI observations. The results show the performance of DNIMAIAC is better than that of DNINSRDB with smaller RMSE by 0.5-1.5%, smaller positive mean bias by 0.8-3.1% and comparable correlation for the 3 sites analyzed. Overall, 1-km MAIAC AOD shows higher accuracy than 2-km elevation-based MERRA-2 AOD, leading to better performance of the simulated DNI using MAIAC AOD. Therefore, 1-km MAIAC AOD can be used to improve the accuracy of solar resource assessment.

14 SOLAR ENERGY↗

Scaling Structured Multigrid to 500K+ Cores Through Coarse-Grid Redistribution

The effcient solution of sparse, linear systems resulting from the discretization of partial differential equations is crucial to the performance of many physics-based simulations. The algorithmic optimality of multilevel approaches for common discretizations makes them a good candidate for an efficient parallel solver. Yet, modern architectures for high-performance computing systems continue to challenge the parallel scalability of multilevel solvers. While algebraic multigrid methods are robust for solving a variety of problems, the increasing importance of data locality and cost of data movement in modern architectures motivates the need to carefully exploit structure in the problem. Robust logically structured variational multigrid methods, such as Black Box Multigrid (BoxMG), maintain structure throughout the multigrid hierarchy. This avoids indirection and increased coarse- grid communication costs typical in parallel algebraic multigrid. Nevertheless, the parallel scalability of structured multigrid is challenged by coarse-grid problems where the overhead in communication dominates computation. In this paper, an algorithm is introduced for redistributing coarse-grid problems through incremental agglomeration. Guided by a predictive performance model, this algorithm provides robust redistribution decisions for structured multilevel solvers. A two-dimensional diffusion problem is used to demonstrate the significant gain in performance of this algorithm over the previous approach that used agglomeration to one processor. In addition, the parallel scalability of this approach

97 MATHEMATICS AND COMPUTING↗

Electricity Baseline 2022 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2022 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilized the appdirs Python dependency (https://pypi.org/project/appdirs/). This submission includes the background data used to generate the 2022 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: `python -c "import appdirs; print(appdirs.user_data_dir())"`). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2022 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; data inventory↗

Electricity Baseline 2021 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2021 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2021 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2021 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗

Electricity Baseline 2020 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2020 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2020 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2020 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗

Correlation Processing Of Local Seismic Data: Applications for Autonomous Sensor Deployments

Excavation and operation of an underground facility is likely to produce an extensive suite of seismic signals observable at the surface for perhaps several km. Probably a large fraction of such signals will be correlated, so the design of a monitoring framework should include consideration of a correlation processing capability. Correlation detectors have been shown to be significantly more sensitive than beam-forming power detectors. Although correlation detectors have a limited detection footprint, they can be generalized into multi-rank subspace detectors which are sensitive over a much larger range of source mechanisms and positions. Production of subspace detectors can be automated, so their use in an autonomous framework may be contemplated. Waveform correlation also can be used to produce very high precision phase picks which may be jointly inverted to simultaneously relocate groups of events. The relative precision of the resulting hypocenters is sufficient to visualize structural detail at a scale of less than a few tens of meters. Three possible correlation processor systems are presented. All use a subspace signal detection framework. The simplest system uses a single-component sensor and is capable of detection and classification of signals. The most complicated system uses many sensors deployed around the facility, and is capable of detection, classification, and high-precision source location. Data from a deep underground mine are presented to demonstrate the applicability of correlation processing to monitoring an underground facility. Although the source region covers an area of about 600m by 580m, all but two of the events form clusters at a threshold of 0.7. All the events could have been detected and classified by the subspace detection framework, and high-precision picks can be computed for all cluster members.

58 GEOSCIENCES↗

Final Report - Facilitating Wind Energy: Addressing Challenges around Visual Impacts, Noise, Credible Data, and Local Benefits through Creative Stakeholder Engagement

The project team consisting of the Consensus Building Institute, Inc., Raab Associates, Ltd., and the MIT-Harvard Program on Negotiation created a model and set of tools for building the capacity of state officials to effectively collaborate with diverse stakeholders in advancing wind development policy formation, wind facility siting, and transmission policy and siting. The model was used to enhance the ability of state officials to advance wind development in their states. Training was delivered in Cambridge, MA, in Spring 2011. The training and associated materials, including a Wind Energy Workbook, website, and simulations, is available for ongoing and widespread dissemination throughout the US.

17 WIND ENERGY↗

Building Agent™ Software (BA) [SWRs 12-05 and 17-21]

Energy interfaces provide a new level of motivational information. Many studies have reported energy savings after dashboards for operators are installed, but these provide little actionable information to the occupant and the occupant cannot report individual comfort. The Building Agent™ (BA), in development at the National Renewable Energy Laboratory (NREL), provides an interface for the occupant to converse with the building control system and aims to actively engage and affect occupant behavior to achieve positive energy outcomes. Energy savings achieved via occupant behavioral changes could be as high as 25%, and could rival savings achieved by technological measures. For example, energy audits often show that more than 50% of the energy used in a building is consumed at night. Occupants leaving equipment and lighting on after they leave work is a large contributing factor. Similar savings opportunities have been identified when occupants were away from their desks 50% of the time, while most of their computers, lights, and other equipment remained powered on. Building Agent aims to create cohesion between building automation system (BAS) data, local measurements, and occupant feedback to provide visualizations, empowering occupants and building engineers to take diagnostic, proactive, energy-saving, and comfort-improving actions. The multilayer BA architecture consists of central server, measurement, and distributed application layers. The server layer aggregates global and localized measurements with occupant feedback to correlate perceived comfort with actual data throughout the building. The measurement layer complements BAS data with localized, environmental condition measurements at workstations. The application layer provides a bidirectional communications mechanism between the building and the occupants that accepts comfort feedback and provides energy performance information. The interface layer also uses three displays to influence building energy use: empathetic and personalized gauges, graphical data, and competitive rankings. Building Agent was implemented at NREL in the Research Support Facility (RSF), a LEED Platinum and Net Zero office building, which has received many awards. The constant feedback from occupants has enabled BA to better enhance the BAS data controls. Continued work is being done on BA to increase visualization of data and enhanced the user experience.

Long, Nicholas↗

Toward Parallel Document Clustering

A key challenge to automated clustering of documents in large text corpora is the high cost of comparing documents in a multimillion dimensional document space. The Anchors Hierarchy is a fast data structure and algorithm for localizing data based on a triangle inequality obeying distance metric, the algorithm strives to minimize the number of distance calculations needed to cluster the documents into “anchors” around reference documents called “pivots”. We extend the original algorithm to increase the amount of available parallelism and consider two implementations: a complex data structure which affords efficient searching, and a simple data structure which requires repeated sorting. The sorting implementation is integrated with a text corpora “Bag of Words” program and initial performance results of end-to-end a document processing workflow are reported.

99 GENERAL AND MISCELLANEOUS↗

Scalable access to shared files in a distributed system

Logic may pre-populate cache of compute nodes allocated to a user's session. Logic may access an image file of a global filesystem, the image file to comprise a first set of one or more files associated with a user session and a second set of one or more files upon which execution of the first set will depend. Logic may store the image file in local data storage of each compute node of a spanning tree of compute nodes allocated for the user session. Logic may unpack the image file based on the data. Logic may configure the local environment of each compute node in the spanning tree to identify a location in local memory of each of the files. And logic may initiate execution of a local filesystem service in each of the compute nodes of the spanning tree.

Oganezov, Alexander↗

Multithreaded Algorithms for Graph Coloring

Graph algorithms are challenging to parallelize when high performance and scalability are primary goals. Low concurrency, poor data locality, irregular access pattern, and high data access to computation ratio are among the chief reasons for the challenge. The performance implication of these features is exasperated on distributed memory machines. More success is being achieved on shared-memory, multi-core architectures supporting multithreading. We consider a prototypical graph problem, coloring, and show how a greedy algorithm for solving it can be e*ectively parallelized on multithreaded architectures. We present in particular two di*erent parallel algorithms. The first relies on speculation and iteration, and is suitable for any shared-memory, multithreaded system. The second uses data ow principles and is targeted at the massively multithreaded Cray XMT system. We benchmark the algorithms on three di*erent platforms and demonstrate scalable runtime performance. In terms of quality of solution, both algorithms use nearly the same number of colors as the serial algorithm.

Catalyurek, Umit V↗

Asynchronous in situ connected-components analysis for complex fluid flows

Fluid flow in porous media is at the heart of problems such as groundwater contamination and carbon sequestration, and presents an important challenge for scientific computing. For this class of problem, large three-dimensional simulations are performed to advance scientific inquiry. On massively parallel computing systems, the volume of data generated by such approaches can become a productivity bottleneck if the raw data generated from the simulation is analyzed in a post-processing step. We present a physics-based framework for in situ data reduction that is theoretically grounded in multiscale averaging theory. We show how task parallelism can be exploited to concurrently perform a variety of analysis tasks with data-dependent costs, including the generation of iso-surfaces, morphological analyses, and connected components analysis. A task management framework is constructed to launch asynchronous analysis threads, manage dependencies between different tasks, promote data locality and hide the impact of data transfers. The framework is applied to analyze GPU-based simulations of two-fluid flow in porous media, generating a set of averaged measures that represents the overall system behavior. We demonstrate how the approach can be applied to perform physically-consistent averaging over fluid subregions using connected components analysis. Simulations performed on Oak Ridge National Lab s Titan supercomputer are profiled to demonstrate the performance of the associated multi-threaded in situ analysis for typical production simulation of two-fluid flow.

Mcclure, James↗

Asynchronous In Situ Connected-Components Analysis for Complex Fluid flows, In: ISAV '16 Proceedings of the 2nd Workshop on In Situ Infrastructures for Enabling Extreme-scale Analysis and Visualization

The simulation of multiscale physics is an important challenge for scientific computing. For this class of problem, large three-dimensional simulations are performed to advance scientific inquiry. On massively parallel computing systems, the volume of data generated by such approaches can become a productivity bottleneck if the raw data generated from the simulation is analyzed in a post-processing step. To address this, we present a physics-based framework for in situ data reduction that is theoretically grounded in multiscale averaging theory. We show how task parallelism can be exploited to concurrently perform a variety of analysis tasks with data-dependent costs, including the generation of iso-surfaces, morphological analyses, and connected components analysis. All analyses are performed in parallel using distributed memory and use the same domain decomposition as the simulation. A task management framework is constructed to leverage available parallelism within a node for analysis. The capabilities of the framework are to launch asynchronous analysis threads, manage dependencies between different tasks, promote data locality and minimize the impact of data transfers. The framework is applied to analyze GPU-based simulations of two-fluid-phase flow in porous media, generating a set of averaged measures that represents the overall system behavior. We demonstrate how the approach can be applied to perform physically-consistent analysis over fluid sub-regions determined from connected components analysis. Simulations performed on Oak Ridge National Lab's Titan supercomputer are profiled to demonstrate the performance of the associated multi-threaded in situ analysis approach for typical production simulation of two-fluid-phase flow.

Computer Science↗