Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data locality”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35
All sourcesNASA NTRSDOE OSTI
NASA NTRS1987

Local concurrent error detection and correction in data structures using virtual backpointers

A new technique, based on virtual backpointers, for local concurrent error detection and correction in linked data structures is presented. Two new data structures, the Virtual Double Linked List, and the B-tree with Virtual Backpointers, are described. For these structures, double errors can be detected in 0(1) time and errors detected during forward moves can be corrected in 0(1) time. The application of a concurrent auditor process to data structure error detection and correction is analyzed, and an implementation is described, to determine the effect on mean time to failure of a multi-user shared database system. The implementation utilizes a Sequent shared memory multiprocessor system operating on a shared databased of Virtual Double Linked Lists.

Li, C. C.↗
NASA NTRS1989

Local concurrent error detection and correction in data structures using virtual backpointers

A new technique, based on virtual backpointers, for local concurrent error detection and correction in linked data strutures is presented. Two new data structures, the Virtual Double Linked List, and the B-tree with Virtual Backpointers, are described. For these structures, double errors can be detected in 0(1) time and errors detected during forward moves can be corrected in 0(1) time. The application of a concurrent auditor process to data structure error detection and correction is analyzed, and an implementation is described, to determine the effect on mean time to failure of a multi-user shared database system. The implementation utilizes a Sequent shared memory multiprocessor system operating on a shared database of Virtual Double Linked Lists.

Li, Chung-Chi Jim↗
DOE OSTI2014

MO-E-17A-01: BEST IN PHYSICS (IMAGING) - Calculating SSDE From CT Exams Using Size Data Available in the DICOM Header of CT Localizer Radiographs

Purpose: To demonstrate the feasibility of using existing data stored within the DICOM header of certain CT localizer radiographs as a patient size metric for calculating CT size-specific dose estimates (SSDE). Methods: For most Siemens CT scanners, the CT localizer radiograph (topogram) contains a private DICOM field that stores an array of numbers describing AP and LAT attenuation-based measures of patient dimension. The square root of the product of the AP and LAT size data, which provides an estimate of water-equivalent-diameter (WED), was calculated retrospectively from topogram data of 20 patients who received clinically-indicated abdomen/pelvis (n=10) and chest (n=10) scans (WED-topo). In addition, slice-by-slice water-equivalent-diameter (WED-image) and effective diameter (ED-image) values were calculated from the respective image data. Using TG-204 lookup tables, size-dependent conversion factors were determined based upon WED-topo, WED-image and ED-image values. These conversion factors were used with the reported CTDIvol to calculate slice-by-slice SSDE for each method. Averaging over all slices, a single SSDE value was determined for each patient and size metric. Patientspecific SSDE and CTDIvol values were then compared with patientspecific organ doses derived from detailed Monte Carlo simulations of fixed tube current scans. Results: For abdomen/pelvis scans, the average difference between liver dose and CTDIvol, SSDE(WED-topo), SSDE(WED-image), and SSDE(ED-image) was 18.70%, 8.17%, 6.84%, and 7.58%, respectively. For chest scans, the average difference between lung dose and CTDIvol, SSDE(WED-topo), SSDE(WED-image), and SSDE(ED-image) was 25.80%, 3.33%, 4.11%, and 7.66%, respectively. Conclusion: SSDE calculated using WED derived from data in the DICOM header of the topogram was comparable to SSDE calculated using WED and ED derived from axial images; each of these estimated organ dose to within 10% for both abdomen/pelvis and chest CT examinations. The topogrambased method has the advantage that WED data are already provided and therefore available without additional post-processing of the image data. Funding Support: NIH Grant R01-EB017095; Disclosures - Michael McNitt-Gray: Institutional Research Agreement, Siemens AG; Research Support, Siemens AG; Consultant, Flaherty Sensabaugh Bonasso PLLC; Consultant, Fulbright and Jaworski; Disclosures - Cynthia McCollough: Research Grant, Siemens Healthcare.

60 APPLIED LIFE SCIENCES↗
NASA NTRS1980

The soft X-ray diffuse background and the structure of the local interstellar medium

Data from five rocket flights of the Wisconsin soft X-ray sky survey, covering the southern galactic hemisphere with 6.5 deg spatial resolution between 0.1 and 2.0 keV, are presented as intensity maps. Analysis of the data in two energy bands below 0.3 keV is presented here. These low-energy X-rays appear to be emitted from a plasma of about one-million K, the bulk of which is closer to the sun than even the nearest 5 x 10 to the 19th/sq cm of neutral gas. No evidence is found for the presence of nearby small (radius of about 0.6 pc) clouds of cool neutral gas, as has been proposed by McKee and Ostriker (1977). The X-ray measurements are consistent with the small amounts of local neutral material inferred from Lyman-alpha absorption measurements.

Fried, P. M.↗
DOE OSTI2025

Electrical Load Forecasting Over Multihop Smart Metering Networks With Federated Learning

Electric load forecasting is essential for power management and stability in smart grids. This is mainly achieved via advanced metering infrastructure, where smart meters (SMs) record household energy data. Traditional machine learning (ML) methods are often employed for load forecasting, but require data sharing, which raises data privacy concerns. Federated learning (FL) can address this issue by running distributed ML models at local SMs without data exchange. However, current FL-based approaches struggle to achieve efficient load forecasting due to imbalanced data distribution across heterogeneous SMs. Here, this article presents a novel personalized FL (PFL) method for high-quality load forecasting in metering networks. A meta-learning-based strategy is developed to address data heterogeneity at local SMs in the collaborative training of local load forecasting models. Moreover, to minimize the load forecasting delays in our PFL model, we study a new latency optimization problem based on optimal resource allocation at SMs. A theoretical convergence analysis is also conducted to provide insights into FL design for federated load forecasting. Extensive simulations from real-world datasets show that our method outperforms existing approaches regarding better load forecasting and reduced operational latency costs.

Rahman, Ratun [Univ. of Alabama, Huntsville, AL (U↗
NASA NTRS1986

Infrasonic emissions from local meteorological events: A summary of data taken throughout 1984

Records of infrasonic signals, propagating through the Earth's atmosphere in the frequency band 2 to 16 Hz, were gathered on a three microphone array at Langley Research Center throughout the year 1984. Digital processing of these records fulfilled three functions: time delay estimation, based on an adaptive filter; source location, determined from the time delay estimates; and source identification, based on spectral analysis. Meteorological support was provided by significant meteorological advisories, lightning locator plots, and daily reports from the Air Weather Service. The infrasonic data are organized into four characteristic signatures, one of which is believed to contain emissions from local meteorological sources. This class of signature prevailed only on those days when major global meteorological events appeared in or near to eastern United States. Eleven case histories are examined. Practical application of the infrasonic array in a low level wing shear alert system is discussed.

Zuckerwar, A. J.↗
DOE OSTI2017

ATLAS (Automatic Tool for Local Assembly Structures) - A Comprehensive Infrastructure for Assembly, Annotation, and Genomic Binning of Metagenomic and Metaranscripomic Data

ATLAS (Automatic Tool for Local Assembly Structures) is a comprehensive multiomics data analysis pipeline that is massively parallel and scalable. ATLAS contains a modular analysis pipeline for assembly, annotation, quantification and genome binning of metagenomics and metatranscriptomics data and a framework for reference metaproteomic database construction. ATLAS transforms raw sequence data into functional and taxonomic data at the microbial population level and provides genome-centric resolution through genome binning. ATLAS provides robust taxonomy based on majority voting of protein coding open reading frames rolled-up at the contig level using modified lowest common ancestor (LCA) analysis. ATLAS provides robust taxonomy based on majority voting of protein coding open reading frames rolled-up at the contig level using modified lowest common ancestor (LCA) analysis. ATLAS is user-friendly, easy install through bioconda maintained as open-source on GitHub, and is implemented in Snakemake for modular customizable workflows.

genome binning↗
NASA NTRS1979

Impact of Short Interval SMS Digital Data on Wind Vector Determination for a Severe Local Storms Area

The impact of 5 minute interval SMS-2 visible digital image data in analyzing severe local storms is examined using wind vectors derived from cloud tracking on time lapsed sequence of geosynchronous satellite images. The cloud tracking areas are located in the Central Plains, where on 6 May 1975, hail-producing thunderstorms occurred ahead of a well defined dry line. The results demonstrate that satellite-derived wind vectors and their associated divergence fields complement conventional meteorological analyses in describing the conditions preceding severe local storm development.

Peslen, C. A.↗
DOE OSTI2015

Engineering the CernVM-Filesystem as a High Bandwidth Distributed Filesystem for Auxiliary Physics Data

A common use pattern in the computing models of particle physics experiments is running many distributed applications that read from a shared set of data files. We refer to this data is auxiliary data, to distinguish it from (a) event data from the detector (which tends to be different for every job), and (b) conditions data about the detector (which tends to be the same for each job in a batch of jobs). Relatively speaking, conditions data also tends to be relatively small per job where both event data and auxiliary data are larger per job. Unlike event data, auxiliary data comes from a limited working set of shared files. Since there is spatial locality of the auxiliary data access, the use case appears to be identical to that of the CernVM- Filesystem (CVMFS). However, we show that distributing auxiliary data through CVMFS causes the existing CVMFS infrastructure to perform poorly. We utilize a CVMFS client feature called 'alien cache' to cache data on existing local high-bandwidth data servers that were engineered for storing event data. This cache is shared between the worker nodes at a site and replaces caching CVMFS files on both the worker node local disks and on the site's local squids. We have tested this alien cache with the dCache NFSv4.1 interface, Lustre, and the Hadoop Distributed File System (HDFS) FUSE interface, and measured performance. In addition, we use high-bandwidth data servers at central sites to perform the CVMFS Stratum 1 function instead of the low-bandwidth web servers deployed for the CVMFS software distribution function. We have tested this using the dCache HTTP interface. As a result, we have a design for an end-to-end high-bandwidth distributed caching read-only filesystem, using existing client software already widely deployed to grid worker nodes and existing file servers already widely installed at grid sites. Files are published in a central place and are soon available on demand throughout the grid and cached locally on the site with a convenient POSIX interface. This paper discusses the details of the architecture and reports performance measurements.

Dykstra, D.↗
DOE OSTI2017

Exploring performance and energy tradeoffs for irregular applications: A case study on the Tilera many-core architecture

High performance, parallel applications with irregular data accesses are becoming a critical workload class for modern systems. In particular, the execution of such workloads on emerging many-core systems is expected to be a significant component of applications in data mining, machine learning, scientific computing and graph analytics. However, power and energy constraints limit the capabilities of individual cores, memory hierarchy and on-chip interconnect of such systems, thus leading to architectural and software trade-os that must be understood in the context of the intended application’s behavior. Irregular applications are notoriously hard to optimize given their data-dependent access patterns, lack of structured locality and complex data structures and code patterns. We have ported two irregular applications, graph community detection using the Louvain method (Grappolo) and high-performance conjugate gradient (HPCCG), to the Tilera many-core system and have conducted a detailed study of platform-independent and platform-specific optimizations that improve their performance as well as reduce their overall energy consumption. To conduct this study, we employ an auto-tuning based approach that explores the optimization design space along three dimensions - memory layout schemes, GCC compiler flag choices and OpenMP loop scheduling options. We leverage MIT’s OpenTuner auto-tuning framework to explore and recommend energy optimal choices for different combinations of parameters. We then conduct an in-depth architectural characterization to understand the memory behavior of the selected workloads. Finally, we perform a correlation study to demonstrate the interplay between the hardware behavior and application characteristics. Using auto-tuning, we demonstrate whole-node energy savings and performance improvements of up to 49:6% and 60% relative to a baseline instantiation, and up to 31% and 45:4% relative to manually optimized variants.

auto tuning↗
DOE OSTI2024

A Framework for Compressing Unstructured Scientific Data via Serialization

We present a general framework for compressing unstructured scientific data with known local connectivity. A common application is simulation data defined on arbitrary finite element meshes. The framework employs a greedy topology preserving reordering of original nodes which allows for seamless integration into existing data processing pipelines. This reordering process depends solely on mesh connectivity and can be performed offline for optimal efficiency. However, the algorithm’s greedy nature also supports on-the-fly implementation. The proposed method is compatible with any compression algorithm that leverages spatial correlations within the data. The effectiveness of this approach is demonstrated on a large-scale real dataset using several compression methods, including MGARD, SZ, and ZFP.

Reshniak, Viktor [ORNL] (ORCID:0000000315454462)↗
DOE OSTI2015

Optimizing Irregular Applications for Energy and Performance on the Tilera Many-core Architecture

Optimizing applications simultaneously for energy and performance is a complex problem. High performance, parallel, irregular applications are notoriously hard to optimize due to their data-dependent memory accesses, lack of structured locality and complex data structures and code patterns. Irregular kernels are growing in importance in applications such as machine learning, graph analytics and combinatorial scientific computing. Performance- and energy-efficient implementation of these kernels on modern, energy efficient, multicore and many-core platforms is therefore an important and challenging problem. We present results from optimizing two irregular applications { the Louvain method for community detection (Grappolo), and high-performance conjugate gradient (HPCCG) { on the Tilera many-core system. We have significantly extended MIT's OpenTuner auto-tuning framework to conduct a detailed study of platform-independent and platform-specific optimizations to improve performance as well as reduce total energy consumption. We explore the optimization design space along three dimensions: memory layout schemes, compiler-based code transformations, and optimization of parallel loop schedules. Using auto-tuning, we demonstrate whole node energy savings of up to 41% relative to a baseline instantiation, and up to 31% relative to manually optimized variants.

irregular applications, energy opttimization, many↗
DOE OSTI2019

Visualization and Analytics of Distribution Systems with Deep Penetration of Distributed Energy Resources (VADER)

In its current state, the distribution system is incapable of handling small to moderate amounts of photovoltaic (PV) penetration. This is because it was initially designed for handling passive loads which, at the level of a substation, have low variability and are forecastable with high accuracy. It has been an open loop system with little monitoring and control. With the addition of PV energy sources, the overall scenario will change dramatically due to (1) two way power flow on network; and (2) high aggregate variability. Additionally, changes on the consumption side lead to a number of smart loads, Electric Vehicles (EVs), and Demand Response. These fundamental changes in the characteristics of the generation and consumption of power will lead to a number of practical engineering problems which must be overcome to allow increased penetration of Distributed PV. Solving the unique engineering challenges which arise at moderate levels of PV penetration requires closed loop integration of data from (1) PV sources; (2) customer load data from smart meters; (3) EV charging data; and (4) local and line mounted precision instruments. These data are not traditionally used by utilities in operations since they are ”non-SCADA” and the current grid does not require such levels of control. To integrate this data and provide real time intelligence from these non-SCADA data, we created the Visualization and Analytics of Distribution Systems with Deep Penetration of Distributed Energy Resources (VADER) platform. VADER is a collection of analytics enabled by integration of massive and heterogeneous data streams for granular real-time monitoring, visualization and control of Distributed Energy Resources (DER) in distribution networks. VADER analytics enable utilities to have greater visibility into distributed energy resources. We built several batch- and stream-analytics in VADER which help operators better understand the impact of distributed energy resources on the grid.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Microcam: A Low Power and Privacy Preserving Multi-modal Platform for Occupancy Detection (Final Report)

Heating, ventilation, and air conditioning (HVAC) consumes a significant portion of the energy used in buildings. Much of this is wasted energy, used when buildings are either not occupied at all, or occupied well under their maximum design conditions. This project has focused on residential occupancy detection to autonomously control HVAC systems and save energy. Limitations of existing occupancy sensors include one or more of the following: (i) they employ sensors or algorithms that are not able to detect stationary occupants; (ii) they cannot classify the source of the motion (such as a pet); (iii) depending on the camera resolution and employed algorithms, they do not allow for embedded or onboard computation, and require external or cloud-based processing; (iv) many algorithms developed for camera-based systems are sensitive to lighting changes, and thus prone to missed detections or false alarms; (v) Most existing systems depend on adjustment of settings for different scenarios, complicating self-commissioning; (vi) they cannot provide high enough accuracy; (vii) they are costly; (viii) they are not battery-powered, thus limiting ease of use and installation. In this project, Syracuse University and its partner SRI have developed a low-cost, high accuracy, standalone residential occupancy sensing platform, referred to as the MicroCam, to address all of the aforementioned challenges. MicroCam can operate on typical alkaline batteries without relying on the “cloud” or external computing resources, and consists of low-power, Artificial Intelligence (AI)-based, IoT platforms. Each platform has multi-modal sensors and can process motion, audio and video data, and send binary occupancy result to a lead platform. All sensor data is processed locally on platforms, and the only transmitted data is the binary occupancy state. In addition, preliminary work has been done on images wherein occupants are not discernable. Thus, MicroCam is a standalone solution preserving privacy of the occupants.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

IM3 Phase 2 Official Simulations: GCAM-Demeter-SELECT Annualized Land Use and Land Cover, Wood Harvest and Fertilization Data with Dynamic Urbanization Harmonized to CLM Land Definitions at 0.125 Degrees

Annualized land use land cover data, including wood harvest and fertilizer use data from the Global Change Analysis Model (GCAM) downscaled to 0.125 degrees for couping with the Community Land Model (CLM). GCAM here refers to GCAM-USA v5.3.im3 which has an enhanced electricity sector and an updated data system needed to represent regional to local scale dynamics. Data is also harmonized with future urbanization projections from the Spatially-Explicit, Long-term, Empirical City developmenT (SELECT) model. Projections/Data are generated using the demeter land use and land cover downscaling model. Original projections were generated at 0.05 degrees before being aggregared to 0.125 degrees. Projections are available for 8 alternative scenarios. Two versions of final data are included- one with managed forests or harvested forest area per pixel broken out and one with the same aggregated into total forests. Following folders are included: demeter_78_PFT_output:This is the final output of dynamic land use land cover change for 78 PFTs as required by CLM raw_outputs_incl_managed_forest: This is the final output but with managed forests broken out as a different PFT. Essentially a 79th PFT is added. wood_harvest_outputs: Wood harvest output per pixel in gC/m2 fertilization_outputs: Fertilizer use per pixel in gN/m2 Each NetCDF file in each folder represents a projection for a separate year, scenario. Land use outputs are organized as PFT level data saved as subdata. Link to GCAM version used- https://data.msdlive.org/records/yb23g-44274 Link to SELECT documentation -https://www.sciencedirect.com/science/article/pii/S1364815219301707 Link to CLM documentation- - https://www.cesm.ucar.edu/models/clm In case of questions contact- kanishka.narayan@pnnl.gov

GCAM-USA↗
NASA NTRS1987

Kinematics at the intersection of the Garlock and Death Valley fault zones, California: Integration of TM data and field studies

The Garlock and Death Valley fault zones in SE California are two active strike-slip faults coming together on the east side of the Avawatz Mtns. The kinematics of this intersection, and the possible continuation of either fault zone, are being investigated using a combination of field mapping, and processing and interpretation of remotely sensed image data. Regional and local relationships are derivable from Thematic Mapper data (30 m resolution), including discrimination and relative age dating of alluvial fans, bedrock mapping, and fault mapping. Aircraft data provide higher spatial resolution over more limited areas. Hypotheses being considered are: (1) the Garlock fault extends east of the intersection; (2) the Garlock fault terminates at the intersection and the Death Valley fault continues southeastward; and (3) the Garlock fault has been offset right laterally by the Death Valley fault which continues to the southeast. Preliminary work indicates that the first hypothesis is invalid. From kinematic considerations, image analysis, and field work the third hypothesis is favored. The projected continuation of the Death Valley zone defines the boundary between the Mojave crustal block and the Basin and Range block.

Abrams, Michael↗
NASA NTRS2013

Web Application Software for Ground Operations Planning Database (GOPDb) Management

A Web application facilitates collaborative development of the ground operations planning document. This will reduce costs and development time for new programs by incorporating the data governance, access control, and revision tracking of the ground operations planning data. Ground Operations Planning requires the creation and maintenance of detailed timelines and documentation. The GOPDb Web application was created using state-of-the-art Web 2.0 technologies, and was deployed as SaaS (Software as a Service), with an emphasis on data governance and security needs. Application access is managed using two-factor authentication, with data write permissions tied to user roles and responsibilities. Multiple instances of the application can be deployed on a Web server to meet the robust needs for multiple, future programs with minimal additional cost. This innovation features high availability and scalability, with no additional software that needs to be bought or installed. For data governance and security (data quality, management, business process management, and risk management for data handling), the software uses NAMS. No local copy/cloning of data is permitted. Data change log/tracking is addressed, as well as collaboration, work flow, and process standardization. The software provides on-line documentation and detailed Web-based help. There are multiple ways that this software can be deployed on a Web server to meet ground operations planning needs for future programs. The software could be used to support commercial crew ground operations planning, as well as commercial payload/satellite ground operations planning. The application source code and database schema are owned by NASA.

Lanham, Clifton↗