Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DATA INTENSIVE COMPUTING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

New technologies as decision aids for the advancement of ecological risk assessment

Moore's law states that the number of transistors that can be placed on an integrated circuit doubles every two years (Moore, 1975). This has led to a steady increase in the processing power of computers over time, and technology is now enhancing and advancing software and scientific applications, which has enabled computationally intensive methods such as machine learning, data science, modeling, and simulation. The advancement of computers and data-driven algorithms is profoundly impacting people's lives. It is changing the way we work, the way we learn, and the way we interact with the world around us. Here, this editorial will discuss how scientists can benefit from the latest technology advancements and related tools by incorporating them into the ecological risk assessment (ERA) to study ecosystems as a way to create refined assessments and accelerate the turnaround times.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

SharP: Towards Programming Extreme-Scale Systems with Hierarchical Heterogeneous Memory

The pre-exascale systems are expected to have a significant amount of hierarchical and heterogeneous on-node memory, and this trend of system architecture in extreme-scale systems is expected to continue into the exascale era. Along with hierarchical-heterogeneous memory, the system typically has a high-performing network and a compute accelerator. This system architecture is not only effective for running traditional High Performance Computing (HPC) applications (Big-Compute), but also running data-intensive HPC applications and Big-Data applications. As a consequence, there is a growing desire to have a single system serve the needs of both Big-Compute and Big-Data applications. Though the system architecture supports the convergence of the Big-Compute and Big-Data, the programming models have yet to evolve to support either hierarchical-heterogeneous memory systems or the convergence. In this work, we propose and develop the programming abstraction called SHARed data-structure centric Programming abstraction (SharP) to address both of these goals, i.e., provide (1) a simple, usable, and portable abstraction for hierarchical-heterogeneous memory and (2) a unified programming abstraction for Big-Compute and Big-Data applications. To evaluate SharP, we implement a Stencil benchmark using SharP, port QMCPack, a petascale-capable application, and adapt Memcached ecosystem, a popular Big-Data framework, to use SharP, and quantify the performance and productivity advantages. Additionally, we demonstrate the simplicity of using SharP on different memories including DRAM, High-bandwidth Memory (HBM), and non-volatile random access memory (NVRAM).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

A Bloom Filter Based Scalable Data Integrity Check Tool for Large-Scale Dataset

Large scale HPC applications are becoming increasingly data intensive. At Oak Ridge Leadership Computing Facility (OLCF), we are observing the number of files curated under individual project are reaching as high as 200 millions and project data size is exceeding petabytes. These simulation datasets, once validated, often needs to be transferred to archival system for long term storage or shared with the rest of the research community. Ensuring the data integrity of the full dataset at this scale is paramount important but also a daunting task. This is especially true considering that most conventional tools are serial and file-based, unwieldy to use and/or can't scale to meet user's demand.To tackle this particular challenge, this paper presents the design, implementation and evaluation of a scalable parallel checksumming tool, fsum, which we developed at OLCF. It is built upon the principle of parallel tree walk and work-stealing pattern to maximize parallelism and is capable of generating a single, consistent signature for the entire dataset at extreme scale. We also applied a novel bloom-filter based technique in aggregating signatures to overcome the signature ordering requirement. Given the probabilistic nature of bloom filter, we provided a detailed error and trade-off analysis. Using multiple datasets from production environment, we demonstrated that our tool can efficiently handle both very large files as well as many small-file based datasets. Our preliminary test showed that on the same hardware, it outperforms conventional tool by as much as 4×. It also exhibited near-linear scaling properties when provisioned with more compute resources.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Obtaining and Managing Answer Quality for Online Data-Intensive Services

Online data-intensive (OLDI) services use anytime algorithms to compute over large amounts of data and respond quickly. Interactive response times are a priority, so OLDI services parallelize query execution across distributed software components and return best effort answers based on the data so far processed. Omitted data from slow components could lead to better answers, but tracing online how much better the answers could be is difficult. Here, we introduce Ubora, a design method to measure the effect of slow-running components on the quality of answers. Ubora randomly samples online queries and executes them a second time. The first online execution omits data from slow components and provides interactive answers. The second execution uses mature results from intermediate components completed after the online execution finishes. Ubora uses memoization to speed up mature executions by replaying network messages exchanged between components. Our systems-level implementation works for a wide range of services, including Hadoop/Yarn, Apache Lucene, the EasyRec Recommendation Engine, and the OpenEphyra question-answering system. Ubora computes answer quality with more mature executions per second than competing approaches that do not use memoization. With Ubora, we illustrate that answer quality is effective at guiding online admission control. While achieving the same answer quality on high-priority queries, our adaptive controller had 55% higher peak throughput on low-priority queries than a competing controller guided by the rate of timeouts.

answer quality↗
DOE OSTI2023

Green Computing Opportunities & Strategy

Computation is critical to emerging fields of data intensive research, enabling new methodologies, approaches and tools. As the rate of hardware efficiency gains slows, computational time and energy costs increase. To match pace with computation demand, new approaches are needed to keep the opportunity for impact open. Charles Tripp, lead of the Green Computing Catalyzer, discusses research efforts to improve software efficiency to enable faster, less energy-intensive computing.

algorithmic efficiency↗
DOE OSTI2010

Fast full-wave seismic inversion using source encoding.

Full Wavefield Seismic Inversion (FWI) estimates a subsurface elastic model by iteratively minimizing the difference between observed and simulated data. This process is extremely compute intensive, with a cost on the order of at least hundreds of prestack reverse time migrations. For time-domain and Krylov-based frequency-domain FWI, the cost of FWI is proportional to the number of seismic sources inverted. We have found that the cost of FWI can be significantly reduced by applying it to data processed by encoding and summing individual source gathers, and by changing the encoding functions between iterations. The encoding step forms a single gather from many input source gathers. This gather represents data that would have been acquired from a spatially distributed set of sources operating simultaneously with different source signatures. We demonstrate, using synthetic data, significant cost reduction by applying FWI to encoded simultaneous-source data.

58 GEOSCIENCES↗
DOE OSTI2011

Use of Data Denial Experiments to Evaluate ESA Forecast Sensitivity Patterns

The overall goal of this multi-phased research project known as WindSENSE is to develop an observation system deployment strategy that would improve wind power generation forecasts. The objective of the deployment strategy is to produce the maximum benefit for 1- to 6-hour ahead forecasts of wind speed at hub-height ({approx}80 m). In this phase of the project the focus is on the Mid-Columbia Basin region which encompasses the Bonneville Power Administration (BPA) wind generation area shown in Figure 1 that includes Klondike, Stateline, and Hopkins Ridge wind plants. The Ensemble Sensitivity Analysis (ESA) approach uses data generated by a set (ensemble) of perturbed numerical weather prediction (NWP) simulations for a sample time period to statistically diagnose the sensitivity of a specified forecast variable (metric) for a target location to parameters at other locations and prior times referred to as the initial condition (IC) or state variables. The ESA approach was tested on the large-scale atmospheric prediction problem by Ancell and Hakim 2007 and Torn and Hakim 2008. ESA was adapted and applied at the mesoscale by Zack et al. (2010a, b, and c) to the Tehachapi Pass, CA (warm and cools seasons) and Mid-Colombia Basin (warm season only) wind generation regions. In order to apply the ESA approach at the resolution needed at the mesoscale, Zack et al. (2010a, b, and c) developed the Multiple Observation Optimization Algorithm (MOOA). MOOA uses a multivariate regression on a few select IC parameters at one location to determine the incremental improvement of measuring multiple variables (representative of the IC parameters) at various locations. MOOA also determines how much information from each IC parameter contributes to the change in the metric variable at the target location. The Zack et al. studies (2010a, b, and c), demonstrated that forecast sensitivity can be characterized by well-defined, localized patterns for a number of IC variables such as 80-m wind speed and vertical temperature difference. Ideally, the data assimilation scheme used in the experiments would have been based upon an ensemble Kalman filter (EnKF) that was similar to the ESA method used to diagnose the Mid-Colombia Basin sensitivity patterns in the previous studies. However, the use of an EnKF system at high resolution is impractical because of the very high computational cost. Thus, it was decided to use the three-dimensional variational analysis data assimilation that is less computationally intensive and more economically practical for generating operational forecasts. There are two tasks in the current project effort designed to validate the ESA observational system deployment approach in order to move closer to the overall goal: (1) Perform an Observing System Experiment (OSE) using a data denial approach which is the focus of this task and report; and (2) Conduct a set of Observing System Simulation Experiments (OSSE) for the Mid-Colombia basin region. The results of this task are presented in a separate report. The objective of the OSE task involves validating the ESA-MOOA results from the previous sensitivity studies for the Mid-Columbia Basin by testing the impact of existing meteorological tower measurements on the 0- to 6-hour ahead 80-m wind forecasts at the target locations. The testing of the ESA-MOOA method used a combination of data assimilation techniques and data denial experiments to accomplish the task objective.

17 WIND ENERGY↗
DOE OSTI2015

Data Intensive Scientific Workflows on a Federated Cloud: CRADA Final Report

The Fermilab Scientific Computing Division and the KISTI Global Science Experimental Data Hub Center have built a prototypical large-scale infrastructure to handle scientific workflows of stakeholders to run on multiple cloud resources. The demonstrations have been in the areas of (a) Data-Intensive Scientific Workflows on Federated Clouds, (b) Interoperability and Federation of Cloud Resources, and (c) Virtual Infrastructure Automation to enable On-Demand Services.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
NASA NTRS1982

Ultraviolet continuum absorption /less than about 1000 A/ above the quiet sun transition region

Lyman continuum absorption shortward of 912 A in the quiet sun solar transition region is investigated by combining spectra obtained from the Apollo Telescope Mount experiments on Skylab. The most recent atomic data are used to compute line intensities for lines that fall on both sides of the Lyman limit. Lines of O III, O IV, O V, and S IV are considered. The computed intensity ratios of most lines from O IV, O V, and S IV agree with the experimental ratios to within a factor of 2. However, the discrepancies show no apparent wavelength dependence. From this fact, it is concluded that at least part of the discrepancy between theory and observation for lines of these ions can be accounted for by uncertainties in instrumental calibration and atomic data. However, difficulties remain in reconciling observation and theory, particularly for lines of O III, and one line of S IV. The other recent results of Schmahl and Orrall (1979) are also discussed in terms of newer atomic data.

Doschek, G. A.↗
NASA NTRS2010

An Improved Wake Vortex Tracking Algorithm for Multiple Aircraft

The accurate tracking of vortex evolution from Large Eddy Simulation (LES) data is a complex and computationally intensive problem. The vortex tracking requires the analysis of very large three-dimensional and time-varying datasets. The complexity of the problem is further compounded by the fact that these vortices are embedded in a background turbulence field, and they may interact with the ground surface. Another level of complication can arise, if vortices from multiple aircrafts are simulated. This paper presents a new technique for post-processing LES data to obtain wake vortex tracks and wake intensities. The new approach isolates vortices by defining "regions of interest" (ROI) around each vortex and has the ability to identify vortex pairs from multiple aircraft. The paper describes the new methodology for tracking wake vortices and presents application of the technique for single and multiple aircraft.

Switzer, George F.↗
DOE OSTI2025

Refining Jets for CMS Run 3 using Fast Simulation

As the LHC moves into its high-luminosity phase, the CMS experiment must handle more complex data collected at much higher rates. While the Geant4-based simulation application (FullSim) provides highly accurate simulation to complement real data, FullSim’s intensive consumption of computing resources becomes an increasing liability as the rates increase, while faster tools offer an advantage. The fast MC production application (FastSim) delivers a complete simulation with a factor of 10 speedup over FullSim, but introduces inaccuracies in some observables. A specialized refinement method, Fast Perfekt, employs machine learning to improve the accuracy of FastSim. An initial report of this work focused on the refinement of jet flavor tagging observables. This article presents an update on the refinement, focusing on PUPPI jets with Run 3 data-taking conditions. Refinement is extended to include jet transverse momentum as well as its propagation to missing transverse momentum. A gridbased framework and real-time monitoring system have been developed to facilitate optimization and scaling of the refinement to a large number of target variables.

Güngördü, Açelya Deniz [Istanbul Tech. U.]↗
DOE OSTI2018

SharP Data Constructs: Data Constructs to Enable Data-Centric Computing

Extreme-scale applications (i.e., Big-Compute) are becoming increasingly data-intensive, i.e., producing and consuming increasingly large amounts of data. The HPC systems traditionally used for these applications are now used for Big-Data applications such as data analytics, social network analysis, machine learning, and genomics. As a consequence of these trends, the system architecture should be flexible and data-centric. This can already be witnessed in the pre-exascale systems with TBs of on-node hierarchical and heterogeneous memories, PBs of system memory, low-latency, high-throughput networks, and many threaded cores. As such, the pre-exascale systems suit the needs of both Big-Compute and Big-Data applications. Though the system architecture is flexible enough to support both Big-Compute and Big-Data, we argue there is a software gap. Particularly, we need data-centric abstractions to leverage the full potential of the system, i.e., there is a need for native support for data resilience, the ability to express data locality and affinity, mechanisms to reduce data movement, the ability to share data, and abstractions to express User's data usage and data access patterns. In this paper, we (i) show the need for taking a holistic approach towards data-centric abstractions, (ii) show how these approaches were realized in the SHARed data-structure centric Programming abstraction (SharP) library, a data-structure centric programming abstraction, and (iii) apply these approaches to a variety of applications that demonstrate its usefulness. Particularly, we apply these approaches to QMCPack and the Graph500 benchmark and demonstrate the advantages of this approach on extreme-scale systems.

Aderholdt, William Ferrol↗
DOE OSTI2023

PROTEUS: Machine Learning Driven Resilience for Extreme-scale Systems

The objective of this project is to design, develop, and evaluate scalable software to enhance resilience, data checkpointing, program restart, and analysis. The proposed tasks are to 1) develop scalable machine learning techniques to learn temporal change patterns in a scalable and in-situ manner, and to minimize data movement and maximize learning locally closest to data; 2) design a concise data representation and indexing mechanism to capture the distribution of changes in data that can guarantee point-wise user-defined tolerable errors while reducing the data storage requirements by an order of magnitude or more; 3) develop data reduction techniques as library modules; 4) exploit local SSD for minimizing data movement in storage hierarchy; 5) develop anomaly detection algorithms that can predict corruptions based on learning of emerging patterns; 6) develop software libraries to be incorporated within widely used data formats and APIs; and 7) evaluate the proposed software using DOE scientific applications. The outcomes of the proposed work are to satisfy many synergistic data reduction and resilience requirements for large-scale data intensive applications executed on extreme-scale computing systems. The developed mechanism for error-bound data approximation is directly applicable to existing scientific applications. Through machine learning from historical events and change distribution, this work will enable anomaly detection for DOE computer facility.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1999

The Estimation Surface Latent Heat Flux Over the Ocean and its Relationship to Marine Atmospheric Boundary Layer (MABL) Structure

A new technique combining active and passive remote sensing instruments for the estimation of surface latent heat flux over the ocean is presented. This synergistic method utilizes aerosol lidar backscatter data, multi-channel infrared radiometer data and microwave scatterometer data acquired onboard the NASA P-3B research aircraft during an extended field campaign over the Atlantic ocean in support of the Lidar In-space Technology Experiment (LITE) in September of 1994. The 10 meter wind speed derived from the scatterometers and the lidar-radiometer inferred near-surface moisture are used to obtain an estimate of the surface flux of moisture via bulk aerodynamic formulae. The results are compared with the Special Sensor Microwave Imager (SSM/I) daily average latent heat flux and show reasonable agreement. However, the SSM/I values are biased high by about 30 W/sq m. In addition, the MABL height, entrainment zone thickness and integrated lidar backscatter intensity are computed from the lidar data and compared with the magnitude of the surface fluxes. The results show that the surface latent heat flux is most strongly correlated with entrainment zone top, bottom and the integrated MABL lidar backscatter, with corresponding correlation coefficients of 0.62, 0.67 and 0.61, respectively.

Palm, Stephen P.↗
NASA NTRS2001

The Estimation of Surface Latent Heat Flux over the Ocean and its Relationship to Marine Atmospheric Boundary Layer (MABL) Structure

A new technique combining active and passive remote sensing instruments for the estimation of surface latent heat flux over the ocean is presented. This synergistic method utilizes aerosol lidar backscatter data, multi-channel infrared radiometer data, and microwave scatterometer data acquired onboard the NASA P-313 research aircraft during an extended field campaign over the Atlantic ocean in support of the Lidar In-space Technology Experiment (LITE) in September of 1994. The 10 meter wind speed derived from scatterometers and lidar-radiometer inferred near-surface moisture are used to obtain an estimate of the surface flux of moisture via a bulk aerodynamic formula. The results are compared with the Special Sensor Microwave Imager (SSM/I) daily average latent heat flux and show reasonable agreement. However, the SSM/I values are biased low by about 15 W/sq m. In addition, the Marine Atmospheric Boundary Layer (MABL) height, entrainment zone thickness and integrated lidar backscatter intensity are computed from the lidar data and compared with the magnitude of the surface fluxes. The results show that the surface latent heat flux is most strongly correlated with entrainment zone depth, MABL height and the integrated MABL lidar backscatter, with corresponding correlation coefficients of 0.39, 0.43 and 0.71, respectively.

Palm, Stephen P.↗
NASA NTRS2000

The Estimation of Surface Latent Heat Flux Over the Ocean and its Relationship to Marine Atmospheric Boundary Layer (MABL) Structure

A new technique combining active and passive remote sensing instruments for the estimation of surface latent heat flux over the ocean is presented. This synergistic method uses aerosol lidar backscatter data, multi-channel infrared radiometer data and microwave scatterometer data acquired onboard the NASA P-3B research aircraft during an extended field campaign over the Atlantic ocean in support of the Lidar In-space Technology Experiment (LITE) in September of 1994. The 10 meter wind speed derived from the scatterometers and the lidar-radiometer inferred near-surface moisture are used to obtain an estimate of the surface flux of moisture via bulk aerodynamic formulae. The results are compared with the Special Sensor Microwave Imager (SSM/I) daily average latent heat flux and show reasonable agreement with an rms error and bias of about 50 and 25 W per square meters, respectively. In addition, the MABL height, entrainment zone thickness and integrated lidar backscatter intensity are computed from the lidar data and compared with the magnitude of the surface fluxes. The results show that the surface latent heat flux is most strongly correlated with entrainment zone top, bottom and the integrated MABL lidar backscatter, with corresponding correlation coefficients of 0.62, 0.67 and 0.61, respectively.

Palm, Stephen P.↗
DOE OSTI2017

Active Subspace Methods for Data-Intensive Inverse Problems

The project has developed theory and computational tools to exploit active subspaces to reduce the dimension in statistical calibration problems. This dimension reduction enables MCMC methods to calibrate otherwise intractable models. The same theoretical and computational tools can also reduce the measurement dimension for calibration problems that use large stores of data.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2006

Application of the Convolution Formalism to the Ocean Tide Potential: Results from the Gravity and Recovery and Climate Experiment (GRACE)

A computationally efficient approach to reducing omission errors in ocean tide potential models is derived and evaluated using data from the Gravity Recovery and Climate Experiment (GRACE) mission. Ocean tide height models are usually explicitly available at a few frequencies, and a smooth unit response is assumed to infer the response across the tidal spectrum. The convolution formalism of Munk and Cartwright (1966) models this response function with a Fourier series. This allows the total ocean tide height, and therefore the total ocean tide potential, to be modeled as a weighted sum of past, present, and future values of the tide-generating potential. Previous applications of the convolution formalism have usually been limited to tide height models, but we extend it to ocean tide potential models. We use luni-solar ephemerides to derive the required tide-generating potential so that the complete spectrum of the ocean tide potential is efficiently represented. In contrast, the traditionally adopted harmonic model of the ocean tide potential requires the explicit sum of the contributions from individual tidal frequencies. It is therefore subject to omission errors from neglected frequencies and is computationally more intensive. Intersatellite range rate data from the GRACE mission are used to compare convolution and harmonic models of the ocean tide potential. The monthly range rate residual variance is smaller by 4-5%, and the daily residual variance is smaller by as much as 15% when using the convolution model than when using a harmonic model that is defined by twice the number of parameters.

models↗