Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2019

Astrophysical S -factors, Thermonuclear Rates, and Electron Screening Potential for the 3 He(d,p) 4 He Big Bang Reaction via a Hierarchical Bayesian Model

Here, we developed a hierarchical Bayesian framework to estimate S-factors and thermonuclear rates for the 3 He(d,p) 4 He reaction, which impacts the primordial abundances of 3 He and 7 Li. The available data are evaluated and all direct measurements are taken into account in our analysis for which we can estimate separate uncertainties for systematic and statistical effects. For the nuclear reaction model, we adopt a single-level, two-channel approximation of R-matrix theory, suitably modified to take the effects of electron screening at lower energies into account. In addition to the usual resonance parameters (resonance location and reduced widths for the incoming and outgoing reaction channel), we include the channel radii and boundary condition parameters in the fitting process. Our new analysis of the 3 He(d,p) 4 He S-factor data results in improved estimates for the thermonuclear rates. This work represents the first nuclear rate evaluation using R-matrix theory embedded into a hierarchical Bayesian framework, properly accounting for all known sources of uncertainty. Therefore, it provides a test bed for future studies of more complex reactions.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models

In the age of big data, deep learning has emerged as a powerful tool to extract insight and exploit its value, both in industry and scientific applications. One common pattern emerging in such applications is frequent checkpointing of the state of the learning model during training, needed in a variety of scenarios: analysis of intermediate states to explain features and correlations with training data, exploration strategies involving alternative models that share a common ancestor, knowledge transfer, resilience, etc. However, with increasing size of the learning models and popularity of distributed data-parallel training approaches, simple checkpointing techniques used so far face several limitations: low serialization performance, blocking I/O, stragglers due to the fact that only a single process is involved in checkpointing. This paper proposes a checkpointing technique specifically designed to address the aforementioned limitations, introducing efficient asynchronous techniques to hide the overhead of serialization and I/O, and distribute the load over all participating processes. Experiments with two deep learning applications (CANDLE and ResNet) on a pre-Exascale HPC platform (Theta) shows significant improvement over state-of-art, both in terms of checkpointing duration and runtime overhead.

checkpointing↗
DOE OSTI2017

Use of DAGMan in CRAB3 to improve the splitting of CMS user jobs

CRAB3 is a workload management tool used by CMS physicists to analyze data acquired by the Compact Muon Solenoid (CMS) detector at the CERN Large Hadron Collider (LHC). Research in high energy physics often requires the analysis of large collections of files, referred to as datasets. The task is divided into jobs that are distributed among a large collection of worker nodes throughout the Worldwide LHC Computing Grid (WLCG). Splitting a large analysis task into optimally sized jobs is critical to efficient use of distributed computing resources. Jobs that are too big will have excessive runtimes and will not distribute the work across all of the available nodes. However, splitting the project into a large number of very small jobs is also inefficient, as each job creates additional overhead which increases load on infrastructure resources. Currently this splitting is done manually, using parameters provided by the user. However the resources needed for each job are difficult to predict because of frequent variations in the performance of the user code and the content of the input dataset. As a result, dividing a task into jobs by hand is difficult and often suboptimal. In this work we present a new feature called “automatic splitting” which removes the need for users to manually specify job splitting parameters. We discuss how HTCondor DAGMan can be used to build dynamic Directed Acyclic Graphs (DAGs) to optimize the performance of large CMS analysis jobs on the Grid. We use DAGMan to dynamically generate interconnected DAGs that estimate the processing time the user code will require to analyze each event. This is used to calculate an estimate of the total processing time per job, and a set of analysis jobs are run using this estimate as a specified time limit. Some jobs may not finish within the alloted time; they are terminated at the time limit, and the unfinished data is regrouped into smaller jobs and resubmitted.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2017

Parallel Multivariate Spatio-Temporal Clustering of Large Ecological Datasets on Hybrid Supercomputers

A proliferation of data from vast networks of remote sensing platforms (satellites, unmanned aircraft systems (UAS), airborne etc.), observational facilities (meteorological, eddy covariance etc.), state-of-the-art sensors, and simulation models offer unprecedented opportunities for scientific discovery. Unsupervised classification is a widely applied data mining approach to derive insights from such data. However, classification of very large data sets is a complex computational problem that requires efficient numerical algorithms and implementations on high performance computing (HPC) platforms. Additionally, increasing power, space, cooling and efficiency requirements has led to the deployment of hybrid supercomputing platforms with complex architectures and memory hierarchies like the Titan system at Oak Ridge National Laboratory. The advent of such accelerated computing architectures offers new challenges and opportunities for big data analytics in general and specifically, large scale cluster analysis in our case. Although there is an existing body of work on parallel cluster analysis, those approaches do not fully meet the needs imposed by the nature and size of our large data sets. Moreover, they had scaling limitations and were mostly limited to traditional distributed memory computing platforms. We present a parallel Multivariate Spatio-Temporal Clustering (MSTC) technique based on k-means cluster analysis that can target hybrid supercomputers like Titan. We developed a hybrid MPI, CUDA and OpenACC implementation that can utilize both CPU and GPU resources on computational nodes. We describe performance results on Titan that demonstrate the scalability and efficacy of our approach in processing large ecological data sets.

Sreepathi, Sarat↗
DOE OSTI2012

An Assessment of Geological Carbon Storage Options in the Illinois Basin: Validation Phase

The Midwest Geological Sequestration Consortium (MGSC) assessed the options for geological carbon dioxide (CO 2 ) storage in the 155,400 km 2 (60,000 mi 2 ) Illinois Basin, which underlies most of Illinois, western Indiana, and western Kentucky. The region has annual CO 2 emissions of about 265 million metric tonnes (292 million tons), primarily from 122 coal-fired electric generation facilities, some of which burn almost 4.5 million tonnes (5 million tons) of coal per year (U.S. Department of Energy, 2010). Validation Phase (Phase II) field tests gathered pilot data to update the Characterization Phase (Phase I) assessment of options for capture, transportation, and storage of CO 2 emissions in three geological sink types: coal seams, oil fields, and saline reservoirs. Four small-scale field tests were conducted to determine the properties of rock units that control injectivity of CO 2 , assess the total storage resources, examine the security of the overlying rock units that act as seals for the reservoirs, and develop ways to control and measure the safety of injection and storage processes. The MGSC designed field test operational plans for pilot sites based on the site screening process, MVA program needs, the selection of equipment related to CO 2 injection, and design of a data acquisition system. Reservoir modeling, computational simulations, and statistical methods assessed and interpreted data gathered from the field tests. Monitoring, Verification, and Accounting (MVA) programs were established to detect leakage of injected CO 2 and ensure public safety. Public outreach and education remained an important part of the project; meetings and presentations informed public and private regional stakeholders of the results and findings. A miscible (liquid) CO 2 flood pilot project was conducted in the Clore Formation sandstone (Mississippian System, Chesterian Series) at Mumford Hills Field in Posey County, southwestern Indiana, and an immiscible CO 2 flood pilot was conducted in the Jackson sandstone (Mississippian System Big Clifty Sandstone Member) at the Sugar Creek Field in Hopkins County, western Kentucky. Up to 12% incremental oil recovery was estimated based on these pilots. A CO 2 huff puff (HNP) pilot project was conducted in the Cypress Sandstone in the Loudon Field. This pilot was designed to measure and record data that could be used to calibrate a reservoir simulation model. A pilot project at the Tanquary Farms site in Wabash County, southeastern Illinois, tested the potential storage of CO 2 in the Springfield Coal Member of the Carbondale Formation (Pennsylvanian System), in order to gauge the potential for large-scale CO 2 storage and/or enhanced coal bed methane recovery from Illinois Basin coal beds. The pilot results from all four sites showed that CO 2 could be injected into the subsurface without adversely affecting groundwater. Additionally, hydrocarbon production was enhanced, giving further evidence that CO 2 storage in oil reservoirs and coal beds offers an economic advantage. Results from the MVA program at each site indicated that injected CO 2 did not leave the injection zone. Topical reports were completed on the Middle and Late Devonian New Albany Shale and Basin CO 2 emissions. The efficacy of the New Albany Shale as a storage sink could be substantial if low injectivity concerns can be alleviated. CO 2 emissions in the Illinois Basin were projected to be dominated by coal-fired power plants.

20 FOSSIL-FUELED POWER PLANTS↗
DOE OSTI2023

Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images

Abstract The rise of automation and machine learning (ML) in electron microscopy has the potential to revolutionize materials research through autonomous data collection and processing. A significant challenge lies in developing ML models that rapidly generalize to large data sets under varying experimental conditions. We address this by employing a cycle generative adversarial network (CycleGAN) with a reciprocal space discriminator, which augments simulated data with realistic spatial frequency information. This allows the CycleGAN to generate images nearly indistinguishable from real data and provide labels for ML applications. We showcase our approach by training a fully convolutional network (FCN) to identify single atom defects in a 4.5 million atom data set, collected using automated acquisition in an aberration-corrected scanning transmission electron microscope (STEM). Our method produces adaptable FCNs that can adjust to dynamically changing experimental variables with minimal intervention, marking a crucial step towards fully autonomous harnessing of microscopy big data.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2024

CMOS-Based Single-Cycle in-Memory XOR/XNOR

Big data applications are on the rise, and so is the number of data centers. The ever-increasing massive data pool needs to be periodically backed up in a secure environment. Moreover, a massive amount of securely backed-up data is required for training binary convolutional neural networks for image classification. XOR and XNOR operations are essential for large-scale data copy verification, encryption, and classification algorithms. The disproportionate speed of existing compute and memory units makes the von Neumann architecture inefficient to perform these Boolean operations. Compute-in-memory (CiM) has proved to be an optimum approach for such bulk computations. The existing CiM-based XOR/XNOR techniques either require multiple cycles for computing or add to the complexity of the fabrication process. Here, we propose a CMOS-based hardware topology for single-cycle in-memory XOR/XNOR operations. Our design provides at least 2× improvement in the latency compared with other existing CMOS-compatible solutions. We verify the proposed system through circuit/system-level simulations and evaluate its robustness using a 5000-point Monte Carlo variation analysis. This all-CMOS design paves the way for practical implementation of CiM XOR/XNOR at scaled technology nodes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

From calibration to parameter learning: Harnessing the scaling effects of big data in geoscientific modeling

Abstract The behaviors and skills of models in many geosciences (e.g., hydrology and ecosystem sciences) strongly depend on spatially-varying parameters that need calibration. A well-calibrated model can reasonably propagate information from observations to unobserved variables via model physics, but traditional calibration is highly inefficient and results in non-unique solutions. Here we propose a novel differentiable parameter learning (dPL) framework that efficiently learns a global mapping between inputs (and optionally responses) and parameters. Crucially, dPL exhibits beneficial scaling curves not previously demonstrated to geoscientists: as training data increases, dPL achieves better performance, more physical coherence, and better generalizability (across space and uncalibrated variables), all with orders-of-magnitude lower computational cost. We demonstrate examples that learned from soil moisture and streamflow, where dPL drastically outperformed existing evolutionary and regionalization methods, or required only ~12.5% of the training data to achieve similar performance. The generic scheme promotes the integration of deep learning and process-based models, without mandating reimplementation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

SHIELDS Final Technical Report

Predicting variations in the near-Earth space environment that can lead to spacecraft damage and failure, i.e. “space weather”, remains a big space physics challenge. A new capability was developed at Los Alamos National Laboratory (LANL) to understand, model, and predict Space Hazards Induced near Earth by Large Dynamic Storms, the SHIELDS framework. This framework simulates the dynamics of the Surface Charging Environment (SCE), the hot (keV) electrons representing the source and seed populations for the radiation belts, on both macro- and micro-scale. In addition to using physics-based models (like RAM-SCB, BATS-R-US, and iPIC3D), new data assimilation techniques employing data from LANL instruments on the Van Allen Probes and geosynchronous satellites were developed. An order of magnitude improvement in the accuracy in the simulation of the spacecraft surface charging environment was thus obtained. SHIELDS also includes a post-processing tool designed to calculate the surface charging for specific spacecraft geometry using the Curvilinear Particle-In-Cell (CPIC) code and to evaluate anomalies' relation to SCE dynamics. Such diagnostics is critically important when performing forensic analyses of space-system failures.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Infrastructure-Relevant Storms of the Last Century

Historical extreme storm events are widely used to make Probable Maximum Precipitation (PMP) estimates, which form the cornerstone of large water management infrastructure safety. Past studies suggest that extreme precipitation processes can be sensitive to land surface feedback and the planetary warming trend, which makes the future safety of large infrastructures questionable given the projected changes in land cover and temperature in the coming decades. In this study, a numerical modeling framework was employed to reconstruct 10 extreme storms over CONUS that occurred during the past 100 years, which are used by the engineering profession for PMP estimation for large infrastructures such as dams. Results show that the correlation in daily rainfall for such reconstruction can range between 0.4 and 0.7, while the correlation for maximum 3-day accumulation (a standard period used in infrastructure design) is always above 0.5 for post-1948 storms. This suggests that current numerical modeling and reanalysis data allow us to reconstruct big storms after 1948 with acceptable accuracy. For storms prior to 1948, however, reconstruction of storms shows inconsistency with observations. Our study indicates that numerical modeling and data may not have advanced to a sufficient level to understand how such old storms (pre-1948) may behave in future warming and land cover conditions. However, the infrastructure community can certainly rely on the use of model reconstructed extreme storms of the 1948-present period to reassess safety of our large water infrastructures under assumed changes in temperature and land cover.

Chen, Xiaodong↗
DOE OSTI2026

AmeriFlux FLUXNET-1F US-xSP NEON Soaproot Saddle (SOAP)

This is the AmeriFlux Management Project (AMP) created FLUXNET-1F version of the carbon flux data for the site US-xSP NEON Soaproot Saddle (SOAP). This is the FLUXNET version of the carbon flux data for the site US-xSP NEON Soaproot Saddle (SOAP) produced by applying the standard ONEFlux (1F) software. Site Description - The Soaproot Saddle is a complex terrain of coarse hills, steep slopes and narrow drainages. With an elevation of 3274 - 4537’ this site encompasses 1438 acres of mixed conifer forests that are experiencing high levels of mortality due to native Pine beetles. At the core of this site stands a 171’ tall flux tower that collects physical and chemical properties of atmosphere and related process. Soaproot Saddle also hosts an array of sensor measurements along with field observations collected by highly trained NEON staff. The automated instrument measurements and some of the terrestrial observational safor this field site are colocted with NEON's aquatic site, Upper Big Creek, which is located just north of Soaproot Saddle's site boundaries.

Network), NEON (National Ecological Observatory [N↗
DOE OSTI2019

Initiation and Development of Wetlands in Southern Florida Karst Landscape Associated With Accumulation of Organic Matter and Vegetation Evolution

Biological processes can exert important controls on geomorphic evolution in karst landscapes, because carbonate mineral dissolution is augmented and spatially focused by the production of CO 2 and biogenic acids from organic matter (OM) decomposition. In Big Cypress National Preserve (BICY) in southwest Florida (USA), depressional wetlands embedded in a mosaic of pine uplands exhibit regular patterning. To understand the impact of OM decomposition as well as different OM sources on carbonate dissolution and landscape patterning, we measured bulk OM property and biomarker proxies (fatty acids and lignin phenols) in different zones (center vs. edge) of the depressional wetlands. We also constructed age profiles of sediment accretion and pattern genesis using 210Pb activities and compound- specific radiocarbon analysis (CSRA) of long-chain fatty acids. Bulk OM and biomarker data indicate rapid OM accumulation in wetland centers resulting from high primary production and near permanent inundation, in contrast to negligible OM accumulation at wetland edges. Age discontinuities suggest episodic periods of near complete OM loss due to decomposition and/or fire. Based on CSRA, landscape patterning likely initiated in the mid-to-late Holocene, with wetlands at slight higher elevation initiated earlier than wetlands at lower elevation. Bulk stable isotope and lignin phenol biomarkers indicate wetland vegetation was initially dominated by herbaceous species and shifted to woody dominated vegetation as landscape hydrology adjusted to changing precipitation. The rate of weathering is likely impacted by hydrological conditions, vegetation dynamics and environmental perturbations like precipitation and fires, suggesting that this karst landscape is a manifestation of complex ecosystem feedbacks.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Using the optimal combined index weight ratio to improve the probability of anomaly detection in big area additive manufacturing

Big Area Additive Manufacturing (BAAM) of composites requires significant time, energy, and material, so it is critical to reduce production inefficiencies to make functional parts without multiple iterations. Statistical process control coupled with Principal Component Analysis (PCA) is a powerful technique that provides a quick, computationally inexpensive, and intuitive way for operators to detect defects that form in a manufacturing process without massive datasets. Recently, a combined index that is a weighted sum of the Hotelling's T 2 and squared residual error statistics has been proposed that can be monitored in one chart, improving interpretation accuracy and simplicity. However, the literature does not offer a formal method to optimise the weights. Here, we introduce two new approaches to the traditional weight selection approach using simulated and BAAM image data. Approach 1 uses a theoretically motivated optimum inspired by probabilistic principal component analysis. Approach 2 systematically varies the ratio of the weights to find the optimum. We show that approach 1 delivers optimal anomaly detection performance in select cases while approach 2 fares better in practice. Surprisingly, we also show that choosing a more complex PCA model has a minimal negative impact on anomaly detection performance compared to a more simplistic model.

3-dimensional printing↗
DOE OSTI2019

Parsl: Pervasive Parallel Programming in Python

High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.

parallel programming↗
DOE OSTI2012

Toward Real Time Data Analysis for Smart Grids

This paper describes the architecture and design of a novel system for supporting large-scale real-time data analysis for future power grid systems. The widespread deployment of renewable generation, smart grid controls, energy storage, plug-in hybrids, and new conducting materials will require fundamental changes in the operational concepts and principal components of the grid. As a result, the whole system becomes highly dynamic and requires constant adjusting based on real time data. Even though millions of sensors such as phase measurement units (PMU) and smart meters are being widely deployed, a data layer that can analyze this amount of data in real time is needed. Unlike the data fabric in other cloud services, the data layer for smart grids has some unique design requirements. First, this layer must provide real time guarantees. Second, this layer must be scalable to allow a large number of applications to access the data from millions of sensors in real time. Third, reliability is critical and this layer must be able to continue to provide service in face of failures. Fourth, this layer must be secure. We address these challenges though a scalable system architecture that integrates the I/O and data processing capability in a devise set of devices. Data process operations can be placed anywhere from sensors, data storage devices, to control centers. We further employ compression to improve performance. We design a lightweight compression customized for power grid data. Our system can reduce end-to-end response time by reduce I/O overhead through compression and overlap compression operations with I/O. The initial prototype of our system was demonstrated with several use cases from PNNL’s FPGI and show that our system can provide real time guarantees to a diverse set of applications.

Data Analytics, Power Grid, Big Data↗
DOE OSTI2018

Assessing relevance of tweets for risk communication

Although Twitter is used for emergency management activities, the relevance of tweets during a hazard event is still open to debate. Here in this study, six different computational (i.e. Natural Language Processing) and spatiotemporal analytical approaches were implemented to assess the relevance of risk information extracted from tweets obtained during the 2013 Colorado flood event. Primarily, tweets containing information about the flooding events and its impacts were analysed. Examination of the relationships between tweet volume and its content with precipitation amount, damage extent, and official reports revealed that relevant tweets provided information about the event and its impacts rather than any other risk information that public expects to receive via alert messages. However, only 14% of the geo-tagged tweets and only 0.06% of the total fire hose tweets were found to be relevant to the event. By providing insight into the quality of social media data and its usefulness to emergency management activities, this study contributes to the literature on quality of big data. Future research in this area would focus on assessing the reliability of relevant tweets for disaster related situational awareness.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Homomorphic data compression for real time photon correlation analysis

The construction of highly coherent X-ray sources, combined with next-generation detectors that are larger and faster, has enabled new research opportunities across the scientific landscape. Among the techniques that benefit most from these advancements is X-ray photon correlation spectroscopy (XPCS), where faster acquisition unlocks the ability to study faster dynamics within samples. However, faster acquisition on larger detectors also introduces unprecedented challenges for online data processing and offline data storage. Such challenges are particularly prominent for XPCS, where real time analyses require simultaneous calculation of all the previously acquired data in the time series. We present a homomorphic compression scheme to effectively reduce the computational time and memory space required for XPCS analysis. Leveraging similarities in the mathematical expression between a matrix-based compression algorithm and the correlation calculation, our approach allows direct operation on the compressed data without their decompression. The offline compression scheme extends storage capacity by a factor of 40 while preserving key features in the lossy compressed data. Meanwhile, the online compression scheme reduces the computational time to below 1 ms, enabling real time calculation of the correlation functions at kHz framerate. Our demonstration of a homomorphic compression of scientific data provides an effective solution to the big data challenge at coherent light sources. Beyond the example shown in this work, the framework can be extended to facilitate real-time operations directly on a compressed data stream for other techniques.

36 MATERIALS SCIENCE↗