Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

Scalable Data Management of the Uintah Simulation Framework for Next-Generation Engineering Problems with Radiation

The need to scale next-generation industrial engineering problems to the largest computational platforms presents unique challenges. This paper focuses on data management related problems faced by the Uintah simulation framework at a production scale of 260K processes. Uintah provides a highly scalable asynchronous many-task runtime system, which in this work is used for the modeling of a 1000 megawatt electric (MWe) ultra-supercritical (USC) coal boiler. At 260K processes, we faced both parallel I/O and visualization related challenges, e.g., the default file-per-process I/O approach of Uintah did not scale on Mira. In this paper we present a simple to implement, restructuring based parallel I/O technique. We impose a restructuring step that alters the distribution of data among processes. The goal is to distribute the dataset such that each process holds a larger chunk of data, which is then written to a file independently. This approach finds a middle ground between two of the most common parallel I/O schemes–file per process I/O and shared file I/O–in terms of both the total number of generated files, and the extent of communication involved during the data aggregation phase. To address scalability issues when visualizing the simulation data, we developed a lightweight renderer using OSPRay, which allows scientists to visualize the data interactively at high quality and make production movies. Finally, this work presents a highly efficient and scalable radiation model based on the sweeping method, which significantly outperforms previous approaches in Uintah, like discrete ordinates. The integrated approach allowed the USC boiler problem to run on 260K CPU cores on Mira.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Simple and Scalable Streaming: The GRETA Data Pipeline

The Gamma Ray Energy Tracking Array (GRETA) is a state of the art gamma-ray spectrometer being built at Lawrence Berkeley National Laboratory to be first sited at the Facility for Rare Isotope Beams (FRIB) at Michigan State University. A key design requirement for the spectrometer is to perform gamma-ray tracking in near real time. To meet this requirement we have used an inline, streaming approach to signal processing in the GRETA data acquisition system, using a GPU-equipped computing cluster. The data stream will reach 480 thousand events per second at an aggregate data rate of 4 gigabytes per second at full design capacity. We have been able to simplify the architecture of the streaming system greatly by interfacing the FPGA-based detector electronics with the computing cluster using standard network technology. A set of highperformance software components to implement queuing, flow control, event processing and event building have been developed, all in a streaming environment which matches detector performance. Prototypes of all high-performance components have been completed and meet design specifications.

Cromaz, Mario↗
NASA NTRS1993

Cost effective data system design approach for EOS AM-1

The question of how to design a cost-effective data system for a space-based application with a high aggregate data rate is addressed. The steps used by the EOS AM-1 design team are defined. A summary of the design as of January 1993 is outlined. The AM-1 Science Data system is comprised of one subsystem that ingests, formats, and routes data - the Science Formatting Equipment (SFE), and another that records the data. Key interfaces are the interfaces to the high rate science instruments, the real-time transmitters, and the internal interface between the SFR and the Solid State Recorder. A cost-effective design approach for the AM-1 data system must take into account impacts on ground systems, data processing facilities, availability of data for real-time users, and future spacecraft designs.

Westmeyer, Paul A.↗
NASA NTRS2013

Hyperspectral Microwave Atmospheric Sounder (HyMAS) Architecture and Design Accommodations

The Hyperspectral Microwave Atmospheric Sounder (HyMAS) is being developed at Lincoln Laboratories and accommodated by the Goddard Space Flight Center for a flight opportunity on a NASA research aircraft. The term "hyperspectral microwave" is used to indicate an all-weather sounding that performs equivalent to hyperspectral infrared sounders in clear air with vertical resolution of approximately 1 km. Deploying the HyMAS equipped scanhead with the existing Conical Scanning Microwave Imaging Radiometer (CoSMIR) shortens the path to a flight demonstration. Hyperspectral microwave is achieved through the use of independent RF antennas that sample the volume of the Earth s atmosphere through various levels of frequencies, thereby producing a set of dense, spaced vertical weighting functions. The simulations proposed for HyMAS 118/183-GHz system should yield surface precipitation rate and water path retrievals for small hail, soft hail, or snow pellets, snow, rainwater, etc. with accuracies comparable to those of the Advanced Technology Microwave Sounder. Further improvements in retrieval methodology (for example, polarization exploitation) are expected. The CoSMIR instrument is a packaging concept re-used on HyMAS to ease the integration features of the scanhead. The HyMAS scanhead will include an ultra-compact Intermediate Frequency Processor (IFP) module that is mounted inside the door to improve thermal management. The IFP is fabricated with materials made of Low-Temperature Co-fired Ceramic (LTCC) technology integrated with detectors, amplifiers, A/D conversion and data aggregation. The IFP will put out 52 channels of 16 bit data comprised of 4-9 channel data streams for temperature profiles and 2-8 channel streams for water vapor. With the limited volume of the existing CoSMIR scanhead and new HyMAS front end components, the HyMAS team at Goddard began preliminary layout work inside the new drum. Importing and re-using models of the shell, the scan head computer, and the slip rings developed for CoSMIR was the starting point. The next step was to modify the antenna faceplate to accommodate the dimensions of the three dual polarization Gaussian Optics Antenna (GOA) assemblies. Two mechanical concepts for the core technology, the hyperspectral IFP, were captured in a design tradeoff. Connector models considered minimum bend radii for the IFP analog connectors. Hyperspectral imaging is accomplished by strategically using a short wavelength intermediate frequency of 18-29 GHz, and thus reducing the size of components in the connection of the front end to the IFP. The SMK (2.92mm) Series connector will lay near the hinge line to minimize its flexing. The digital output of the IFP will use a Serial Peripheral Interface (SPI) that must be accommodated by the scan head computer. To make that computer more reliable, maintainable, and forward compatible with the 52 HyMAS channels, a testbed of the scan head, calibration, and archive computers and the PIC24 microprocessor that resides on the IFP is in development. The computers will be programmed using a new framework application called Interoperable Remote Component (IRC). This software allows flexibility to program computers that communicate with each other and can adapt easily to the emerging HyMAS requirements for data format, algorithms, and graphical user interface (GUI). It is expected that the CoSMIR instrument will cut over to the IRC after it is adapted on an updated CoSMIR testbed.

Hilliard, Lawrence↗
DOE OSTI2026

Predicting Particle-Size Distributions in Fuel-Cell Inks

Prediction of the catalyst-activated carbon particle sizes of fuel-cell inks remains a critical challenge in enhancing the performance and durability of fuel cells. The performance and structural integrity of the catalyst layers in the cell depend not only on the properties of the ionomer, but also on the carbon supports that host the catalyst. To investigate how these carbon aggregate structures form, we present a model that calculates the cooperative size distributions of ionomer and carbon aggregates in various water/alcohol mixtures and compares those results to available experimental data. Aggregation of both the suspended ionomers and the carbon particles is interwoven as the carbon aggregation depends heavily on the ionic strength of its environment, namely protons dissociating from the ionomer’s sulfonic-acid-group side chains. We demonstrate that the surrounding mixed solvent as well as Nafion TM binder concentration strongly influence the degree of aggregation for carbon particles. The validated model provides a framework for exploring aggregation under a wide range of conditions for various electrochemical-technology inks.

Srivastav, Harsh [University of California, Berkel↗
DOE OSTI2023

Secure Data Logging and Processing with Blockchain and Machine Learning (Final Report)

Secure Data Logging and Processing with Blockchain and Machine Learning (ML) research is focused on the development of a platform to securely log and process sensor data in fossil power plants. The platform integrates two emerging technologies, blockchain and ML, and incorporates several innovative mechanisms to ensure the integrity, reliability, and resiliency of power systems. The goal is to protect the power plant from various cyberattacks such as false data injection and denial of service attacks using these technologies. The research goal was enabled by the following Research Project Objectives: 1) Secure authentication and identity verification of sensor nodes, actuators, and other equipment within a network. 2) Development of mechanisms that ensure only data sent by legitimate sensors are accepted and stored in the data repository. 3) Development of data aggregation methodologies using ML / Deep Learning (DL) algorithms to minimize noise / faulty data. 4) Implementation of the blockchain technologies to provide data security using secured IOTA framework & nodes.

20 FOSSIL-FUELED POWER PLANTS↗
NASA NTRS2017

Re-Organizing Earth Observation Data Storage to Support Temporal Analysis of Big Data

The Earth Observing System Data and Information System archives many datasets that are critical to understanding long-term variations in Earth science properties. Thus, some of these are large, multi-decadal datasets. Yet the challenge in long time series analysis comes less from the sheer volume than the data organization, which is typically one (or a small number of) time steps per file. The overhead of opening and inventorying complex, API-driven data formats such as Hierarchical Data Format introduces a small latency at each time step, which nonetheless adds up for datasets with O(10^6) single-timestep files. Several approaches to reorganizing the data can mitigate this overhead by an order of magnitude: pre-aggregating data along the time axis (time-chunking); storing the data in a highly distributed file system; or storing data in distributed columnar databases. Storing a second copy of the data incurs extra costs, so some selection criteria must be employed, which would be driven by expected or actual usage by the end user community, balanced against the extra cost.

data storage↗
DOE OSTI2026

Investigating the Determinants of Household Capabilities Burden During Power Outages: The Case of Winter Storm Uri

Existing research primarily uses census data to identify the vulnerability of communities to hazards. These vulnerability indices provide aggregated data and are not hazard-specific nor well-validated with post-event data. In contrast, our study uses household survey data (n=1065) to understand which Texan households suffered the greatest loss of their capabilities due to power outages and other utility service disruptions during Winter Storm Uri. Inspired by the Capabilities Approach, our measures of burden include the number of household capability types disrupted during the outages (e.g., cooking, heating, refrigeration), the severity of impact for each disrupted capability, and the additional time and financial costs of coping with these disruptions. We perform a clustering analysis, and find two distinct groups in our data, consisting of ‘lesser burden' and ‘heavier burden' households. Results indicate that the households experiencing the heaviest capabilities burden were most likely to experience longer power outages and the loss of water services. They were also more likely to have a Hispanic-Latino household member, lack access to a generator, live in a rented home, have larger households with more young children, fewer adults over 65, lower household incomes, been impacted by the COVID-19 pandemic, and more family characteristics that made life harder. We also fit a logistic regression model to assess the role of outage, household, and community characteristics in predicting differences in capabilities burden. Our results offer insights into enumerating the consequences of utility service disruptions on households, which can inform more targeted and equitable resilience strategies.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

Modifying the Asynchronous Jacobi Method for Data Corruption Resilience

Moving scientific computation from high-performance computing (HPC) and cloud computing (CC) environments to devices on the edge, i.e., physically near instruments of interest, has received tremendous interest in recent years. Such edge computing environments can operate on data in situ, offering enticing benefits over data aggregation to HPC and CC facilities that include avoiding costs of transmission, increased data privacy, and real-time data analysis. Because of the inherent unreliability of edge computing environments, new fault-tolerant approaches must be developed before the benefits of edge computing can be realized. Motivated by algorithm-based fault tolerance, a variant of the asynchronous Jacobi (ASJ) method is developed that achieves resilience to data corruption by rejecting solution approximations from neighbor devices according to a bound derived from convergence theory. Numerical results on a two-dimensional Poisson problem show that the new rejection criterion, along with a novel approximation to the shortest path length on which the criterion depends, restores convergence for the ASJ variant in the presence of certain types data corruption. Numerical results are obtained for when the singular values in the analytic bound are approximated. Additional linear systems are also explored, one with a more dense sparsity pattern and one that includes advection. All results indicate that successful resilience to data corruption depends on whether the bound tightens fast enough to reject corrupted data before the iteration evolution deviates significantly from that predicted by the convergence theory defining the bound. This observation generalizes to future work on algorithm-based fault tolerance for other asynchronous algorithms, including upcoming approaches that leverage Krylov subspaces.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Impact Mapping for Geospatial Reasoning and Decision Making

Objective The reported study evaluated a novel approach to aiding geospatial reasoning and decision making. Background Impact mapping aims to alleviate the cognitive demands of geospatial tasks in part by externalizing data in the form of an integrated decision surface. This is achieved by aggregating data across multiple sources of information and visualizing their combined utility rather than objective measurements or individual utility. Previous research has shown that geospatial decisions improve when aided in this manner, but it remains unknown if dynamic decision making, often plagued by fatigue and anchoring bias, would benefit similarly. Method The experiment implemented a systematic manipulation of the presence of a composite impact map and the number of attributes present in a two-stage disaster relief, resource allocation task to investigate when and how impact mapping is beneficial or deleterious to decision makers. Results The presence of the composite impact map increased the utility of selected sites, increased re-planning decisions, reduced information display views, and reduced workload. Generally, the effect of the composite impact map was greater when participants were asked to evaluate more attributes. Conclusion Composite impact maps appear to improve repeated geospatial reasoning and minimize anchoring bias because they alleviate the cognitive demands otherwise necessary to interpret and maintain information from multiple attributes. Application Data visualization techniques, such as impact mapping, can improve repeated geospatial decision making in environments that include high cognitive demand.

Illingworth, David A.↗
DOE OSTI2021

ShopperWorkerRatio

These data characterize spatial and temporal variation in the ratio between shoppers and workers in public places - points of interest - in the United States between 2019 and 2020. The underlying data on foot traffic to public places is collected by SafeGraph, a commercial data aggregator. We estimate the ratio of shoppers to workers based on recorded visit duration.These data may be useful for understanding how use of public spaces have changed during the Coronavirus pandemic, and more generally for understanding activity patterns in public.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS1993

Mobile and replicated alignment of arrays in data-parallel programs

When a data-parallel language like FORTRAN 90 is compiled for a distributed-memory machine, aggregate data objects (such as arrays) are distributed across the processor memories. The mapping determines the amount of residual communication needed to bring operands of parallel operations into alignment with each other. A common approach is to break the mapping into two stages: first, an alignment that maps all the objects to an abstract template, and then a distribution that maps the template to the processors. We solve two facets of the problem of finding alignments that reduce residual communication: we determine alignments that vary in loops, and objects that should have replicated alignments. We show that loop-dependent mobile alignment is sometimes necessary for optimum performance, and we provide algorithms with which a compiler can determine good mobile alignments for objects within do loops. We also identify situations in which replicated alignment is either required by the program itself (via spread operations) or can be used to improve performance. We propose an algorithm based on network flow that determines which objects to replicate so as to minimize the total amount of broadcast communication in replication. This work on mobile and replicated alignment extends our earlier work on determining static alignment.

Chatterjee, Siddhartha↗
DOE OSTI2014

Collective input/output under memory constraints

Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input/output (I/O) performance. In this study, we introduce a memory-conscious collective I/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intranode and internode layers, and determines I/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I/O can have a significant positive impact on scientific discovery productivity.

collective input/output↗
NASA NTRS1993

The alignment-distribution graph

Implementing a data-parallel language such as Fortran 90 on a distributed-memory parallel computer requires distributing aggregate data objects (such as arrays) among the memory modules attached to the processors. The mapping of objects to the machine determines the amount of residual communication needed to bring operands of parallel operations into alignment with each other. We present a program representation called the alignment distribution graph that makes these communication requirements explicit. We describe the details of the representation, show how to model communication cost in this framework, and outline several algorithms for determining object mappings that approximately minimize residual communication.

Chatterjee, Siddhartha↗
NASA NTRS1993

The alignment-distribution graph

Implementing a data-parallel language such as Fortran 90 on a distributed-memory parallel computer requires distributing aggregate data objects (such as arrays) among the memory modules attached to the processors. The mapping of objects to the machine determines the amount of residual communication needed to bring operands of parallel operations into alignment with each other. We present a program representation called the alignment-distribution graph that makes these communication requirements explicit. We describe the details of the representation, show how to model communication cost in this framework, and outline several algorithms for determining object mappings that approximately minimize residual communication.

Chatterjee, Siddhartha↗
DOE OSTI2022

The ABCflux database: Arctic–boreal CO 2 flux observations and ancillary information aggregated to monthly time steps across terrestrial ecosystems

Past efforts to synthesize and quantify the magnitude and change in carbon dioxide (CO 2 ) fluxes in terrestrial ecosystems across the rapidly warming Arctic–boreal zone (ABZ) have provided valuable information but were limited in their geographical and temporal coverage. Furthermore, these efforts have been based on data aggregated over varying time periods, often with only minimal site ancillary data, thus limiting their potential to be used in large-scale carbon budget assessments. To bridge these gaps, we developed a standardized monthly database of Arctic–boreal CO 2 fluxes (ABCflux) that aggregates in situ measurements of terrestrial net ecosystem CO 2 exchange and its derived partitioned component fluxes: gross primary productivity and ecosystem respiration. The data span from 1989 to 2020 with over 70 supporting variables that describe key site conditions (e.g., vegetation and disturbance type), micrometeorological and environmental measurements (e.g., air and soil temperatures), and flux measurement techniques. Here, we describe these variables, the spatial and temporal distribution of observations, the main strengths and limitations of the database, and the potential research opportunities it enables. In total, ABCflux includes 244 sites and 6309 monthly observations; 136 sites and 2217 monthly observations represent tundra, and 108 sites and 4092 observations represent the boreal biome. The database includes fluxes estimated with chamber (19 % of the monthly observations), snow diffusion (3 %) and eddy covariance (78 %) techniques. The largest number of observations were collected during the climatological summer (June–August; 32 %), and fewer observations were available for autumn (September–October; 25 %), winter (December–February; 18 %), and spring (March–May; 25 %). ABCflux can be used in a wide array of empirical, remote sensing and modeling studies to improve understanding of the regional and temporal variability in CO 2 fluxes and to better estimate the terrestrial ABZ CO 2 budget.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS1984

Selection of a map grid for data analysis and archival

Arguments for selection of a map grid are reiterated by illustrating the quantitative effects on data quality caused by using different grids. It is shown that the use of the rectangular latitude-longitude grid actually degrades data quality, increases stored data volume, and increases the complexity of data manipulation. Use of an equal-area grid is shown to be a proper way to aggregate data, but such grids are thought to be inconvenient. A solution to this dilemma is proposed by showing that the analysis and archival map grids need not be the same. The results of a proper analysis of the data on an equal-area grid can be remapped to the more 'convenient' rectangular latitude-longitude grid without loss of quality.

Rossow, W. B.↗