Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data analytics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Open-Source Data for MAC-POSTS: Mobility Data Analytics Center - Prediction, Optimization, and Simulation Toolkit for Transportation Systems

MAC-POSTS (Mobility Data Analytics Center - Prediction, Optimization, and Simulation toolkit for Transportation Systems) is a toolkit for dynamic transportation network modeling. Developed by the Mobility Data Analytics Center (MAC) at Carnegie Mellon University, this package implements many classic dynamic transportation network models, as well as new models proposed by MAC members. It has served as one building block for many other models and research projects. As such, this package used to be treated as an internal research project of the MAC lab, and admittedly, the code base is messy, and the interface is hard to use. However, we are working hard to make it a generally usable and useful toolkit for dynamic transportation network modeling. We would really appreciate any feedback, comments, suggestions, or criticisms.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Navigating Exascale Operational Data Analytics: From Inundation to Insight

In this paper, we address the challenges in achieving sustainable data-driven efficiency by providing a detailed exploration of the end-to-end operational data analytics (ODA) framework that evolved through two generations of supercomputer systems at the Oak Ridge Leadership Computing Facility (OLCF). This framework addresses large data streams ingested from heavily instrumented HPC environment that accumulates multi-terabytes per day. We outline the multifaceted data life cycle across HPC procurement, operations, and research & development, identifying key obstacles and design decisions that shape effective strategies in building and supporting data pipelines end-to-end. By sharing key insights and lessons learned from our experience, we offer recommendations for the HPC community on enabling sustainable operational data analytics and beyond. Our contributions aim to bridge the gap between potential and real benefits of operational data, guiding future efforts towards integrated and sustainable operational intelligence in high-performance computing environments.

Shin, Woong↗
DOE OSTI2017

Determining appropriate data analytics for transformer health monitoring - 107

Transformers are vital assets for the safe, reliable and cost-effective operation of nuclear power plants. The unexpected failure of a transformer can lead to different consequences ranging from a lack of export capability, with the corresponding economic penalties, to catastrophic failure, with the associated health, safety and economic effects. Condition monitoring techniques examine the health of the transformer periodically, with the aim to identify early indicators of anomalies. However, many transformer failures occur because diagnostic and monitoring models do not identify degraded conditions in time. Therefore, health monitoring is an essential component to transformer life cycle management. Existing tools for transformer health monitoring use traditional dissolved gas analysis based diagnostics techniques. With the advance of prognostics and health management (PHM) applications, we can enhance traditional transformer health monitoring techniques using PHM analytics. The design of an appropriate data analytics system requires a multi-stage design process including: (i) specification of engineering requirements; (ii) characterization of existing data sources and analytics to identify complementary techniques; (iii) development of the functional specification of the analytics suite to formalize its behavior, and finally (iv) deployment, validation, and verification of the functional requirements in the final platform. Accordingly, in this paper we propose a transformer analytics suite which incorporates anomaly detection, diagnostics, and prognostics modules in order to complement existing tools for transformer health monitoring. (authors)

42 ENGINEERING↗
DOE OSTI2020

Predicting Execution Times for Disk-based and In-Situ Parallel Data Analytics (Final Technical Report)

In recent years, there has been a significant amount of interests in in-situ analytics on simulation programs. For a variety of reasons, it is desirable to be able to predict the execution time of an analytics program. At the same time, frameworks such as MapReduce have become popular for scientific data analytics. This paper focuses on developing performance models for predicting execution time of parallel data analytics, with a special emphasis on in-situ analytics. We take two distinct approach towards performance prediction. We first expand SKOPE (a SKeleton framewOrk for Performance Exploration) with performance models for disk data read, cache performance, and page fault penalty. Second, an analytical performance model is also developed. We have evaluated our performance prediction framework as well as the analytical model on three hardware setups with well-known data mining algorithms implemented in three programming paradigms, MapReduce, MATE (a MapReduce-like parallel system with an alternate API for multi-core environments) and Smart (a MapReduce-like framework for in-situ analytics). Results show that our performance prediction framework along with the incorporated performance models are capable of accurately predicting execution times for parallel scientific analytics on different hardware setups.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Considerations on the Use of Custom Accelerators for Big Data Analytics

Accelerators, including Graphic Processing Units (GPUs) for gen- eral purpose computation, many-core designs with wide vector units (e.g., Intel Phi), have become a common component of many high performance clusters. The appearance of more stable and reliable tools tools that can automatically convert code written in high-level specifications with annotations (such as C or C++) to hardware de- scription languages (High-Level Synthesis - HLS), is also setting the stage for a broader use of reconfigurable devices (e.g., Field Pro- grammable Gate Arrays - FPGAs) in high performance system for the implementation of custom accelerators, helped by the fact that new processors include advanced cache-coherent interconnects for these components. In this chapter, we briefly survey the status of the use of accelerators in high performance systems targeted at big data analytics applications. We argue that, although the progress in the use of accelerators for this class of applications has been sig- nificant, differently from scientific simulations there still are gaps to close. This is particularly true for the ”irregular” behaviors exhibited by no-SQL, graph databases. We focus our attention on the limits of HLS tools for data analytics and graph methods, and discuss a new architectural template that better fits the requirement of this class of applications. We validate the new architectural templates by mod- ifying the Graph Engine for Multithreaded System (GEMS) frame- work to support accelerators generated with such a methodology, and testing with queries coming from the Lehigh University Benchmark (LUBM). The architectural template enables better supporting the task and memory level parallelism present in graph methods by sup- porting a new control model and a enhanced memory interface. We show that out solution allows generating parallel accelerators, pro- viding speed ups with respect to conventional HLS flows. We finally draw conclusions and present a perspective on the use of reconfig- urable devices and Design Automation tools for data analytics.

Data Analytics↗
DOE OSTI2020

Guest Editorial: Advanced Data-Analytics for Power System Operation, Control, and Enhanced Situational Awareness

Along with the smart grid development, modern power systems are entering a ‘data-intensive’ era. A vast volume of data from power grids is being collected through advanced sensing and communication technologies, such as smart metering data, phasor measurement data, as well as meteorological data (e.g., wind speed and solar irradiance) related to renewable power generation. Such data contains comprehensive information about the power system covering equipment's health status, power grid's static and dynamic characteristics, renewable power generation, customers’ electricity usage pattern, etc. Therefore, advanced data-analytics techniques are needed to convert such data to knowledge for practical applications. In line with the trend of widespread data-driven applications in power systems, this Special Issue aims to present state-of-the-art research works on advanced data-analytics for power system's operation, control, and situational awareness. There are in total twenty-six papers accepted for publication in this Special Issue through careful peer reviews and revisions. Under the overarching theme of data-driven applications in power systems, the selected papers are broadly categorised into five topics. The summary of every topic is given below. You are, however, strongly encouraged to read the full paper if interested.

Xu, Yan↗
DOE OSTI2019

Materials data analytics for 9% Cr family steel

A materials data analytics (MDA) methodology was developed in this study to evaluate publicly available information on 9% Cr family steel and to handle nonlinear relationships and the sparsity in materials data for this alloy class. The overarching goal is to accelerate the design process as well as to reduce the time and expense associated with qualification testing of new alloys for fossil energy applications. Data entries in the analyzed data set for 82 iron–base alloy compositions, several processing parameters, and results of tensile mechanical tests selected for this study were arranged in 34 columns by 915 rows. While detailed microstructural information was not available, it is assumed that the compositional space for the 9 to 12% Cr steels is limited such that all data entries have a tempered martensitic microstructure during service. Establishing a hierarchy of first–order trends in the publicly available data requires the MDA to filter out the biases. Complexity of the phase transformations and microstructure evolution in the multicomponent alloys (using 21 chemical elements) with major influence on mechanical properties, leads to inefficiency in direct application of unbiased linear regression across the entire data space. To address the nonlinearity, analyses of tensile data were performed in composition–based clusters. Clusters corresponding to moderately frequent patterns and maximized information gain were further refined by using p–norm distance measures, matching the alloy classification groups adopted by industry. Lastly, the evolutionary method of propagating an ensemble of competing cluster–based models proved to be a viable option in dealing with scarce, multidimensional data.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Phasor-Measurement-Unit-Based Data Analytics Using Digital Twin and PhasorAnalytics Software

A major objective of this project was to apply GE’s commercial machine learning and data analytics toolsets to large-scale, real-world, anonymized Phasor Measurement Unit (PMU) datasets in order to extract signatures, correlated and/or causal factors, and precursor patterns associated with significant power system phenomena. The project had a particular emphasis on extraction of insights relevant to asset health monitoring, real-time load modeling and cybersecurity monitoring. Additionally, the team was directed to undertake a comprehensive data quality analysis for the provided datasets and encouraged to estimate the ‘machine-learning readiness’ of the datasets by documenting any major obstacles to the application of commercial machine learning algorithms. To accomplish the aforementioned objectives, the project team’s work centered around the identification of key event signatures and application of the identified event signatures for event detection and event classification. The industry-validated, semi-supervised machine learning strategy employed for event signature identification involved several major tasks, including data-preprocessing, generation of an overabundance of features, normal data identification, normality modeling, and event signature identification through a methodical, quantitative ranking of features in order of relevance to each studied event type. Throughout the project, data quality issues and mitigation techniques were investigated. In this report, insights are provided regarding the readiness of the provided synchrophasor datasets for application of machine learning and data analytics. The methodologies employed for this technical strategy are summarized in this report. With regards to data preprocessing and feature generation, the provided Training and Test Datasets were ingested into GE’s big data environment. Subsequently, the team applied bad data cleansing and data imputation scripts, event detection scripts, and application programming interfaces (APIs) to the datasets for convenient data access. The project team completed development and validation of dozens of physics-based, statistics-based and transformation-based feature functions used for the extraction of over 60 synchrophasor features. Using a new parallel feature generation technology developed on this project, over 60 features have been rapidly generated for the full two years’ worth of Training and Test Dataset data associated with both the Eastern and Western interconnects. Even accommodating for temporal down-sampling inherent to the feature extraction procedure, this parallel feature generation activity resulted in a massive feature set with a storage requirement approximately equal to that of the raw training dataset itself. With regards to normal data identification and normality modeling, a normality model was built using the feature data extracted from the Training Dataset and iteratively refined subsequent to incremental adjustments and expansions of the Training Dataset feature data. With respect to event characterization and signature identification, an event signature identification pipeline was developed and used in conjunction with the normality model to identify over 15 event signatures for key event categories within the Training Dataset. The identified event signatures were used to characterize hundreds of key events in terms of relative severity, duration, and location of the event. An investigation was undertaken to identify correlated and causal factors involved in transformer events. A separate investigation into temporal trends in ring-down analysis results was undertaken to determine possible associations between system dynamics and various other factors such as loading, season or year. To validate the identified event signatures, additional work was undertaken to develop signature-based anomaly detection and classification tools suitable for convenient application to the synchrophasor datasets. The anomaly detection and classification tools, suitable for online application, were then applied to the entirety of the Eastern Interconnect Training and Test Datasets. Performance of the event detection and classification tools was evaluated upon receipt of the Test Dataset event logs (i.e., the labels for events contained in the Test Dataset), and promising results were obtained despite several challenges (documented herein) associated with application of supervised or semi-supervised machine learning methods to large-scale, anonymized datasets. Finally, the detection and classification tools were used to detect, classify, and characterize thousands of new events not included in the original event logs provided by the DOE within both the Training and Test Datasets.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2017

Coupling Big Data Analytics and Reactive Transport Modeling for Cost-effective Groundwater Monitoring - 17163

This study presents an innovative approach for sustainable and cost-effective groundwater monitoring. This approach takes advantage of recent advances in various technologies: (1) in situ autonomous sensors, (2) big data analytics, and (3) parallel high-performance computing for flow and reactive transport modeling. In situ sensors are used to periodically measure the key variables (such as pH, redox potential, electrical conductivity, and groundwater level), which control contaminant mobility and the plume spatial and temporal distribution. Based on a limited number of groundwater sampling, the data analytics methods-data mining and machine learning-allow us to identify and quantify the correlations between the in situ-measured variables and contaminant concentrations, and also to detect significant changes associated with the plume mobility. In addition, a state-of-the-art parallel numerical flow and reactive transport simulator Amanzi, and uncertainty quantification software Agni are used to provide an improved physical and mechanistic understanding of the contaminated groundwater system behavior, and to predict the long-term plume distribution for optimizing and adapting the monitoring strategy. Amanzi and Agni were developed as part of the Advanced Simulation Capability for Environmental Management (ASCEM) program of the DOE Office of Environmental Management. Such modeling is critical, particularly, for assessing the impact of climate change and associated hydrological shifts. The developed approach is expected to significantly reduce the groundwater sampling frequency and associated cost. In addition, the real-time information on plume mobility serves as an early warning system, improving the resiliency of contaminated or potentially contaminated sites. We demonstrate this approach using as an example the Savannah River Site (SRS) F-Area, where groundwater is contaminated by various radionuclides, including uranium, tritium and technetium. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2018

Using Big Data Analytics to Advance Precision Radiation Oncology

Big clinical data analytics as a primary component of precision medicine is discussed, identifying where these emerging tools fit in the spectrum of genomics and radiomics research. A learning health system (LHS) is conceptualized that uses clinically acquired data with machine learning to advance the initiatives of precision medicine. The LHS is comprehensive and can be used for clinical decision support, discovery, and hypothesis derivation. These developing uses can positively impact the ultimate management and therapeutic course for patients. The conceptual model for each use of clinical data, however, is different, and an overview of the implications is discussed. With advancements in technologies and culture to improve the efficiency, accuracy, and breadth of measurements of the patient condition, the concept of an LHS may be realized in precision radiation therapy.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2022

Smoky Mountain Data Challenge 2021: An Open Call to Solve Scientific Data Challenges Using Advanced Data Analytics and Edge Computing

The 2021 Smoky Mountains Computational Sciences and Engineering Conference enlists scientists from across Oak Ridge National Laboratory (ORNL) and industry to be data sponsors and help create data analytics and edge computing challenges for eminent datasets in a variety of scientific domains. This work describes the significance of each of the eight datasets and their associated challenge questions. The challenge questions for each dataset were required to cover multiple difficulty levels. An international call for participation was sent to students, asking them to form teams of up to six people and apply novel data analytics and edge computing methods to solve these challenges.

Devineni, Pravallika↗
DOE OSTI2016

Predictive Big Data Analytics: A Study of Parkinson’s Disease Using Large, Complex, Heterogeneous, Incongruent, Multi-Source and Incomplete Observations

Background A unique archive of Big Data on Parkinson’s Disease is collected, managed and disseminated by the Parkinson’s Progression Markers Initiative (PPMI). The integration of such complex and heterogeneous Big Data from multiple sources offers unparalleled opportunities to study the early stages of prevalent neurodegenerative processes, track their progression and quickly identify the efficacies of alternative treatments. Many previous human and animal studies have examined the relationship of Parkinson’s disease (PD) risk to trauma, genetics, environment, co-morbidities, or life style. The defining characteristics of Big Data–large size, incongruency, incompleteness, complexity, multiplicity of scales, and heterogeneity of information-generating sources–all pose challenges to the classical techniques for data management, processing, visualization and interpretation. We propose, implement, test and validate complementary model-based and model-free approaches for PD classification and prediction. To explore PD risk using Big Data methodology, we jointly processed complex PPMI imaging, genetics, clinical and demographic data. Methods and Findings Collective representation of the multi-source data facilitates the aggregation and harmonization of complex data elements. This enables joint modeling of the complete data, leading to the development of Big Data analytics, predictive synthesis, and statistical validation. Using heterogeneous PPMI data, we developed a comprehensive protocol for end-to-end data characterization, manipulation, processing, cleaning, analysis and validation. Specifically, we (i) introduce methods for rebalancing imbalanced cohorts, (ii) utilize a wide spectrum of classification methods to generate consistent and powerful phenotypic predictions, and (iii) generate reproducible machine-learning based classification that enables the reporting of model parameters and diagnostic forecasting based on new data. We evaluated several complementary model-based predictive approaches, which failed to generate accurate and reliable diagnostic predictions. However, the results of several machine-learning based classification methods indicated significant power to predict Parkinson’s disease in the PPMI subjects (consistent accuracy, sensitivity, and specificity exceeding 96%, confirmed using statistical n-fold cross-validation). Clinical (e.g., Unified Parkinson's Disease Rating Scale (UPDRS) scores), demographic (e.g., age), genetics (e.g., rs34637584, chr12), and derived neuroimaging biomarker (e.g., cerebellum shape index) data all contributed to the predictive analytics and diagnostic forecasting. Conclusions Model-free Big Data machine learning-based classification methods (e.g., adaptive boosting, support vector machines) can outperform model-based techniques in terms of predictive precision and reliability (e.g., forecasting patient diagnosis). We observed that statistical rebalancing of cohort sizes yields better discrimination of group differences, specifically for predictive analytics based on heterogeneous and incomplete PPMI data. UPDRS scores play a critical role in predicting diagnosis, which is expected based on the clinical definition of Parkinson’s disease. Even without longitudinal UPDRS data, however, the accuracy of model-free machine learning based classification is over 80%. The methods, software and protocols developed here are openly shared and can be employed to study other neurodegenerative disorders (e.g., Alzheimer’s, Huntington’s, amyotrophic lateral sclerosis), as well as for other predictive Big Data analytics applications.

Science & Technology - Other Topics↗
DOE OSTI2026

A critical review on additive manufacturing of refractory alloys from a data analytics perspective- beyond nickel-based superalloys

Refractory alloys (RAs) are promising materials due to their exceptional physicochemical properties, but most research remains at the laboratory scale. For broader adoption, advancements in manufacturing are essential. Because their high stability makes conventional methods like machining and casting difficult, additive manufacturing (AM) is emerging as an effective approach for fabricating refractory alloy components. However, AM's repeated non-equilibrium thermal cycles introduce undesired features (e.g. defects, anisotropic microstructures, and residual stresses), which are magnified due to RAs’ unique properties. This paper comprehensively reviews the state-of-the-art methods of AM for refractory alloys. It explores data analytics techniques to establish design rules based on multi-fidelity experimental and computational methods. Furthermore, it investigates integrated, collaborative efforts to harmonise standalone databases, information, knowledge, and predictive models at multi-physics, multi-stage, and multi-scale. Unlike the existing literature that focuses primarily on material systems or process fundamentals, this work provides an integrated perspective on AM of refractory alloys from a data analytics standpoint, highlighting the roles of integrated computational materials engineering (ICME), verification, validation, and uncertainty quantification (VV&UQ), and digital twin-driven qualification in overcoming data scarcity and accelerating rapid qualification.

Additive manufacturing↗
DOE OSTI2015

Scalable and Power Efficient Data Analytics for Hybrid Exascale Systems

This project developed a generic and optimized set of core data analytics functions. These functions organically consolidate a broad constellation of high performance analytical pipelines. As the architectures of emerging HPC systems become inherently heterogeneous, there is a need to design algorithms for data analysis kernels accelerated on hybrid multi-node, multi-core HPC architectures comprised of a mix of CPUs, GPUs, and SSDs. Furthermore, the power-aware trend drives the advances in our performance-energy tradeoff analysis framework which enables our data analysis kernels algorithms and software to be parameterized so that users can choose the right power-performance optimizations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Data Analytics Applied to Coal Fired Boilers for Detecting Leaks

Data analytics were used to detect boiler leaks from five different coal-fired boilers including both subcritical and supercritical systems. Discriminant functions were developed that detected leaks up to two weeks prior to forced plant shutdowns for repairs. The leaks were identified to occur at different sections of the boiler for each plant, including waterwalls, economizer and superheater using conventional process measurement data. Leaking conditions were detected with a high degree of confidence (≪ 1% misclassified observations) and were able to distinguish normal operations from those time periods with steam leaks even while operating the power plants in power cycling mode.Multivariable statistical analyses, including Principal Component (PCA), cluster, and Fischer Discriminant Analysis (FDA) were used to characterize the leak occurrence. Normal and operational states with steam leaks were provided in the original process datasets. These datasets were split into two different groups for training and validation purposes. The data were sorted chronologically, and every third observation was assigned to training the Discriminant Function Model (DFM) while the rest were reserved for validation. PCA was used to reduce dimensionality of the original datasets. Canonical and FDA analyses were used to investigate the relationship between process variables. The outcome of the analyses revealed that nearly 35,000 observations were classified correctly; less than 0.05% of total observations were misclassified to be leaking, i.e. both false positives and false negatives.

Indrawan, Natarianto↗
DOE OSTI2021

Synergistic data analytics for electromechanical oscillation in electric power systems

Accurate real-time estimation of the four electromechanical oscillation properties, i.e., dominant oscillation modes, mode shapes, participation factors, and coherent groups, is of great importance to assess and mitigate potential electromechanical oscillations in interconnected power systems. While eigenvalue analysis can realize such estimation, it requires precise linearized dynamic models and accurate parameters, which are highly difficult to obtain in practice. Data fusion-based modal estimation methods can extract the properties of electromechanical oscillations from measurement data without the power system model and parameters, but most of the time only one or two property assessments can be accomplished each time. To overcome this challenge, this paper presents a synergistic data analytics solution to characterize the dynamic behaviors of electromechanical oscillations from real-time measurement data. Here, the proposed method uses optimized variable projection, and it is capable of estimating all four electromechanical oscillation behavior properties from measured responses. Case studies are performed using the simulated measurement data of a 16-generator 68-bus test system and the field measurements collected by the PMUs deployed in the Yunnan Power Grid. The results demonstrate that the proposed synergistic data analytics solution can achieve satisfactory performance in capturing the properties of electromechanical oscillations from measurement data and exhibit strong robustness against measurement noise when compared with existing measurement-based methods.

42 ENGINEERING↗