Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “big data and analytics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

G-mode magnetic force microscopy: Separating magnetic and electrostatic interactions using big data analytics

In this work, we develop a full information capture approach for Magnetic Force Microscopy (MFM), referred to as generalized mode (G-Mode) MFM. G-Mode MFM acquires and stores the full data stream from the photodetector, captured at sampling rates approaching the intrinsic photodiode limit. The data can be subsequently compressed, denoised, and analyzed, without information loss. Here, G-Mode MFM is implemented and compared to the traditional heterodyne-based MFM on model systems, including domain structures in ferromagnetic Yttrium Iron Garnet and the electronically and magnetically inhomogeneous high entropy alloy, CoFeMnNiSn. We investigate the use of information theory to mine the G-Mode MFM data and demonstrate its usefulness for extracting information which may be hidden in traditional MFM modes, including signatures of nonlinearities and mode-coupling phenomena. Finally, we demonstrate detection and separation of magnetic and electrostatic tip-sample interactions from a single G-Mode image, by analyzing the entire frequency response of the cantilever. G-Mode MFM is immediately implementable on any atomic force microscopy platform and as such is expected to be a useful technique for probing spatiotemporal cantilever dynamics and mapping material properties, as well as their mutual interactions.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log

Reliability, availability and serviceability (RAS) logs of high performance computing (HPC) resources, when closely investigated in spatial and temporal dimensions, can provide invaluable information regarding system status, performance, and resource utilization. These data are often generated from multiple logging systems and sensors that cover many components of the system. The analysis of these data for finding persistent temporal and spatial insights faces two main difficulties: the volume of RAS logs makes manual inspection difficult and the unstructured nature and unique properties of log data produced by each subsystem adds another dimension of difficulty in identifying implicit correlation among recorded events. To address these issues, we recently developed a multi-user Big Data analytics framework for HPC log data at Oak Ridge National Laboratory (ORNL). This paper introduces three in-progress data analytics projects that leverage this framework to assess system status, mine event patterns, and study correlations between user applications and system events. We describe the motivation of each project and detail their workflows using three years of log data collected from ORNL's Titan supercomputer.

Park, Byung↗

Big data analytics in smart grids: state-of-the-art, challenges, opportunities, and future directions

Big data has a potential to unlock novel groundbreaking opportunities in the power grid sector that enhances a multitude of technical, social, and economic gains. The currently untapped potential of applying the science of big data for better planning and operation of the power grid is a very challenging task and needs significant efforts all-around. As power grid technologies evolve in conjunction with measurement and communication technologies, this results in unprecedented amount of heterogeneous big data sets from diverse sources. In particular, computational complexity, data security, and operational integration of big data into utility decision frameworks are the key challenges to transform the heterogeneous large dataset into actionable outcomes. Moreover, due to the complex nature of power grids along with the need to balance power in real time, seamless integration of big data into utility operations is very critical. In this context, big data analytics combined with grid visualization can lead to better situational awareness and predictive decisions. This paper presents a comprehensive state-of-the-art review of big data analytics and its applications in power grids, and also identifies challenges and opportunities from utility, industry, and research perspectives. The paper analyzes research gaps and presents insights on future research directions to integrate big data analytics into electric utility decision framework. Detailed information for utilities looking to apply big data analytics and details insights on how utilities can enhance revenue streams and bring disruptive innovation in the industry are discussed. More importantly, general guidelines for utilities to make the right investment in the adoption of big data analytics by unveiling interdependencies among critical infrastructures and operations are also provided.

13 HYDRO ENERGY↗

Big-Data Analytics for Electric Grid and Demand-Side Management

This report describes research to characterize the status of, and trends in, big-data analytics for the electricity grid. The research focused on 1) identifying power-grid big-data sources, types, and characteristics; and 2) characterizing big-data architecture, analytic methods, technology applications, and challenges. The first part of this report describes the main sources and characteristics of big data for the smart grid and comprehensively reviews big-data architecture, technologies, and applications in the power sector. The second part of this report presents case studies of big data applications in the power industry: (1) a smart-meter data and predictive analytics method for demand response (DR), (2) synchrophasor data analytics for the distribution grid, and (3) utility data for peak-demand management. For the predictive analytics case study, smart-meter-data-driven and physical models were developed to predict the potential kilowatt (kW) capacity reduction from DR. The DR estimation framework that was developed works for both small and large-scale customers. The synchrophasor case study demonstrates use of an algorithm applied to time-series data to detect events that appear as significant changes, known as “edges,” in voltage magnitude measurements. The synchrophasor case study also introduces an approach for clustering sets of events to reveal unique features that distinguish them (e.g., distinguishing capacitor bank switching from transformer tap changes). The peak-demand management case study describes the use of the data analytics to enable DR programs to limit forecasted peak demand, resulting in cost savings to the utility. The findings from the research described in this report support identification of opportunities and technologies for big-data and analytics applications for demand-side management in the power sector as well as other approaches to modernizing the electricity grid.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Using Big Data Analytics to Advance Precision Radiation Oncology

Big clinical data analytics as a primary component of precision medicine is discussed, identifying where these emerging tools fit in the spectrum of genomics and radiomics research. A learning health system (LHS) is conceptualized that uses clinically acquired data with machine learning to advance the initiatives of precision medicine. The LHS is comprehensive and can be used for clinical decision support, discovery, and hypothesis derivation. These developing uses can positively impact the ultimate management and therapeutic course for patients. The conceptual model for each use of clinical data, however, is different, and an overview of the implications is discussed. With advancements in technologies and culture to improve the efficiency, accuracy, and breadth of measurements of the patient condition, the concept of an LHS may be realized in precision radiation therapy.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Application of Systems Engineering Principles and Techniques in Biological Big Data Analytics: A Review

In the past few decades, we have witnessed tremendous advancements in biology, life sciences and healthcare. These advancements are due in no small part to the big data made available by various high-throughput technologies, the ever-advancing computing power, and the algorithmic advancements in machine learning. Specifically, big data analytics such as statistical and machine learning has become an essential tool in these rapidly developing fields. As a result, the subject has drawn increased attention and many review papers have been published in just the past few years on the subject. Different from all existing reviews, this work focuses on the application of systems, engineering principles and techniques in addressing some of the common challenges in big data analytics for biological, biomedical and healthcare applications. Specifically, this review focuses on the following three key areas in biological big data analytics where systems engineering principles and techniques have been playing important roles: the principle of parsimony in addressing overfitting, the dynamic analysis of biological data, and the role of domain knowledge in biological data analytics.

dynamic analysis↗

Predictive Big Data Analytics: A Study of Parkinson’s Disease Using Large, Complex, Heterogeneous, Incongruent, Multi-Source and Incomplete Observations

Background A unique archive of Big Data on Parkinson’s Disease is collected, managed and disseminated by the Parkinson’s Progression Markers Initiative (PPMI). The integration of such complex and heterogeneous Big Data from multiple sources offers unparalleled opportunities to study the early stages of prevalent neurodegenerative processes, track their progression and quickly identify the efficacies of alternative treatments. Many previous human and animal studies have examined the relationship of Parkinson’s disease (PD) risk to trauma, genetics, environment, co-morbidities, or life style. The defining characteristics of Big Data–large size, incongruency, incompleteness, complexity, multiplicity of scales, and heterogeneity of information-generating sources–all pose challenges to the classical techniques for data management, processing, visualization and interpretation. We propose, implement, test and validate complementary model-based and model-free approaches for PD classification and prediction. To explore PD risk using Big Data methodology, we jointly processed complex PPMI imaging, genetics, clinical and demographic data. Methods and Findings Collective representation of the multi-source data facilitates the aggregation and harmonization of complex data elements. This enables joint modeling of the complete data, leading to the development of Big Data analytics, predictive synthesis, and statistical validation. Using heterogeneous PPMI data, we developed a comprehensive protocol for end-to-end data characterization, manipulation, processing, cleaning, analysis and validation. Specifically, we (i) introduce methods for rebalancing imbalanced cohorts, (ii) utilize a wide spectrum of classification methods to generate consistent and powerful phenotypic predictions, and (iii) generate reproducible machine-learning based classification that enables the reporting of model parameters and diagnostic forecasting based on new data. We evaluated several complementary model-based predictive approaches, which failed to generate accurate and reliable diagnostic predictions. However, the results of several machine-learning based classification methods indicated significant power to predict Parkinson’s disease in the PPMI subjects (consistent accuracy, sensitivity, and specificity exceeding 96%, confirmed using statistical n-fold cross-validation). Clinical (e.g., Unified Parkinson's Disease Rating Scale (UPDRS) scores), demographic (e.g., age), genetics (e.g., rs34637584, chr12), and derived neuroimaging biomarker (e.g., cerebellum shape index) data all contributed to the predictive analytics and diagnostic forecasting. Conclusions Model-free Big Data machine learning-based classification methods (e.g., adaptive boosting, support vector machines) can outperform model-based techniques in terms of predictive precision and reliability (e.g., forecasting patient diagnosis). We observed that statistical rebalancing of cohort sizes yields better discrimination of group differences, specifically for predictive analytics based on heterogeneous and incomplete PPMI data. UPDRS scores play a critical role in predicting diagnosis, which is expected based on the clinical definition of Parkinson’s disease. Even without longitudinal UPDRS data, however, the accuracy of model-free machine learning based classification is over 80%. The methods, software and protocols developed here are openly shared and can be employed to study other neurodegenerative disorders (e.g., Alzheimer’s, Huntington’s, amyotrophic lateral sclerosis), as well as for other predictive Big Data analytics applications.

Science & Technology - Other Topics↗

Coupling Big Data Analytics and Reactive Transport Modeling for Cost-effective Groundwater Monitoring - 17163

This study presents an innovative approach for sustainable and cost-effective groundwater monitoring. This approach takes advantage of recent advances in various technologies: (1) in situ autonomous sensors, (2) big data analytics, and (3) parallel high-performance computing for flow and reactive transport modeling. In situ sensors are used to periodically measure the key variables (such as pH, redox potential, electrical conductivity, and groundwater level), which control contaminant mobility and the plume spatial and temporal distribution. Based on a limited number of groundwater sampling, the data analytics methods-data mining and machine learning-allow us to identify and quantify the correlations between the in situ-measured variables and contaminant concentrations, and also to detect significant changes associated with the plume mobility. In addition, a state-of-the-art parallel numerical flow and reactive transport simulator Amanzi, and uncertainty quantification software Agni are used to provide an improved physical and mechanistic understanding of the contaminated groundwater system behavior, and to predict the long-term plume distribution for optimizing and adapting the monitoring strategy. Amanzi and Agni were developed as part of the Advanced Simulation Capability for Environmental Management (ASCEM) program of the DOE Office of Environmental Management. Such modeling is critical, particularly, for assessing the impact of climate change and associated hydrological shifts. The developed approach is expected to significantly reduce the groundwater sampling frequency and associated cost. In addition, the real-time information on plume mobility serves as an early warning system, improving the resiliency of contaminated or potentially contaminated sites. We demonstrate this approach using as an example the Savannah River Site (SRS) F-Area, where groundwater is contaminated by various radionuclides, including uranium, tritium and technetium. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Considerations on the Use of Custom Accelerators for Big Data Analytics

Accelerators, including Graphic Processing Units (GPUs) for gen- eral purpose computation, many-core designs with wide vector units (e.g., Intel Phi), have become a common component of many high performance clusters. The appearance of more stable and reliable tools tools that can automatically convert code written in high-level specifications with annotations (such as C or C++) to hardware de- scription languages (High-Level Synthesis - HLS), is also setting the stage for a broader use of reconfigurable devices (e.g., Field Pro- grammable Gate Arrays - FPGAs) in high performance system for the implementation of custom accelerators, helped by the fact that new processors include advanced cache-coherent interconnects for these components. In this chapter, we briefly survey the status of the use of accelerators in high performance systems targeted at big data analytics applications. We argue that, although the progress in the use of accelerators for this class of applications has been sig- nificant, differently from scientific simulations there still are gaps to close. This is particularly true for the ”irregular” behaviors exhibited by no-SQL, graph databases. We focus our attention on the limits of HLS tools for data analytics and graph methods, and discuss a new architectural template that better fits the requirement of this class of applications. We validate the new architectural templates by mod- ifying the Graph Engine for Multithreaded System (GEMS) frame- work to support accelerators generated with such a methodology, and testing with queries coming from the Lehigh University Benchmark (LUBM). The architectural template enables better supporting the task and memory level parallelism present in graph methods by sup- porting a new control model and a enhanced memory interface. We show that out solution allows generating parallel accelerators, pro- viding speed ups with respect to conventional HLS flows. We finally draw conclusions and present a perspective on the use of reconfig- urable devices and Design Automation tools for data analytics.

Data Analytics↗

Big Data Analytics for Long-Term Meteorological Observations at Hanford Site

A growing number of physical objects with embedded sensors with typically high volume and frequently updated data sets has accentuated the need to develop methodologies to extract useful information from big data for supporting decision making. This study applies a suite of data analytics and core principles of data science to characterize near real-time meteorological data with a focus on extreme weather events. To highlight the applicability of this work and make it more accessible from a risk management perspective, a foundation for a software platform with an intuitive Graphical User Interface (GUI) was developed to access and analyze data from a decommissioned nuclear production complex operated by the U.S. Department of Energy (DOE, Richland, USA). Exploratory data analysis (EDA), involving classical non-parametric statistics, and machine learning (ML) techniques, were used to develop statistical summaries and learn characteristic features of key weather patterns and signatures. The new approach and GUI provide key insights into using big data and ML to assist site operation related to safety management strategies for extreme weather events. Specifically, this work offers a practical guide to analyzing long-term meteorological data and highlights the integration of ML and classical statistics to applied risk and decision science.

54 ENVIRONMENTAL SCIENCES↗

Big Data Analytics for Demand Response: Clustering Over Space and Time

The pervasive deployment of advanced sensing infrastructure in Cyber-Physical systems, such as the Smart Grid, has resulted in an unprecedented data explosion. Such data exhibit both large volumes and high velocity characteristics, two of the three pillars of Big Data, and have a time-series notion as datasets in this context typically consist of successive measurements made over a time interval. Time-series data can be valuable for data mining and analytics tasks such as identifying the “right” customers among a diverse population, to target for Demand Response programs. However, time series are challenging to mine due to their high dimensionality. In this paper, we motivate this problem using a real application from the smart grid domain. We explore novel representations of time-series data for BigData analytics, and propose a clustering technique for determining natural segmentation of customers and identification of temporal consumption patterns. Our method is generizable to large-scale, real-world scenarios, without making any assumptions about the data. We evaluate our technique using real datasets from smart meters, totaling ~ 18,200,000 data points, and show the efficacy of our technique in efficiency detecting the number of optimal number of clusters.

97 MATHEMATICS AND COMPUTING↗

Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography

Electron microscopy is undergoing a transition; from the model of producing only a few micrographs, through the current state where many images and spectra can be digitally recorded, to a new mode where very large volumes of data (movies, ptychographic and multi-dimensional series) can be rapidly obtained. In this paper, we discuss the application of so-called “big-data” methods to high dimensional microscopy data, using unsupervised multivariate statistical techniques, in order to explore salient image features in a specific example of BiFeO 3 domains. Remarkably, k-means clustering reveals domain differentiation despite the fact that the algorithm is purely statistical in nature and does not require any prior information regarding the material, any coexisting phases, or any differentiating structures. While this is a somewhat trivial case, this example signifies the extraction of useful physical and structural information without any prior bias regarding the sample or the instrumental modality. Further interpretation of these types of results may still require human intervention. Finally, however, the open nature of this algorithm and its wide availability, enable broad collaborations and exploratory work necessary to enable efficient data analysis in electron microscopy.

36 MATERIALS SCIENCE↗

Big Data Analytic for Cascading Failure Analysis

With the challenges of increased grid dynamics and more variability of power generation from renewable energy sources, rapidly increasing complexity in the grid model, and abundant data from measurements and simulations, the requirements for computational analysis have also increased dramatically. Power system operators and engineers need to analyze more scenarios, extract meaningful information from a larger set of data, and respond more quickly when faced with these new challenges for large-scale applications, such as contingency analysis. This paper proposes a novel big data analysis approach for power system cascading analysis, prevention, and remediation. The developed techniques will be capable of cascading analysis, better assessment of the system’s vulnerability level, as well as proposing potential remediation. Case studies using IEEE 118-bus system and a 500-bus system, with a comparison against a commercial tool, validate the advantages of the developed big data approach: accurate prediction, and more importantly, faster and effective correction actions.The developed techniques could be further used for other power system applications.

big data, cascading analysis, corrective action, m↗

Big data analytics in the building industry

Catalyzed by recent market, technology, and policy trends, energy data collection in the building industry is becoming more widespread. This wealth of information allows more data-driven decision-making by designers, commissioning agents, facilities staff, and energy service providers during the course of building design, operation and retrofit. The U.S. Department of Energy’s Building Performance Database (BPD) has taken advantage of this wealth of building asset- and energy-related data by collecting, cleansing, and standardizing data from across the U.S. on over 870,00 buildings, and is designed to support building benchmarking, energy efficiency project design, and buildings-related policy development with real-world data. Here, this article explores the promises and perils energy professionals are faced with when leveraging such tools, presenting example analyses for commercial and residential buildings, highlighting potential issues, and discussing solutions and best practices that will enable designers, operators and commissioning agents to make the most of ‘big data’ resources such as the BPD.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗