SEARCH · Engineering Papers

Results for “exploratory data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2024

Visualisation and outlier detection for probability density function ensembles

Abstract Exploratory data analysis (EDA) for functional data—data objects where observations are entire functions—is a difficult problem that has seen significant attention in recent literature. This surge in interest is motivated by the ubiquitous nature of functional data, which are prevalent in applications across fields such as meteorology, biology, medicine and engineering. Empirical probability density functions (PDFs) can be viewed as constrained functional data objects that must integrate to one and be nonnegative. They show up in contexts such as yearly income distributions, zooplankton size structure in oceanography and in connectivity patterns in the brain, among others. While PDF data are certainly common in modern research, little attention has been given to EDA specifically for PDFs. In this paper, we extend several methods for EDA on functional data for PDFs and compare them on simulated data that exhibit different types of variation, designed to mimic that seen in real‐world applications. We then use our new methods to perform EDA on the breakthrough curves observed in gas transport simulations for underground fracture networks.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Parallel sorting algorithm classification: is manual instrumentation necessary?

Understanding parallel algorithms is crucial for accelerating scientific simulations on complex, distributed memory, high-performance computers. Modern algorithm classification approaches learn semantics directly from source code to differentiate between algorithms, however, accessing source code is not always possible. We can learn about parallel algorithms from observing their performance, as programs running the same algorithms and using the same hardware should exhibit similar performance characteristics. We present an approach to learn algorithm classes from parallel performance data directly in order to classify algorithms without access to the source code. We extend previous work to enable classifying parallel sorting algorithms using automatic instrumentation instead of requiring manual region annotations in the source code. In this work, we design and demonstrate a study for classification of parallel sorting algorithms using parallel performance data collected from automatic instrumentation, and evaluate the performance of our new methodology on classification. We leverage Caliper to collect the performance data, Thicket for our exploratory data analysis (EDA), and PyTorch and Scikit-learn to evaluate the effectiveness of random forests, support vector machines (SVMs), decision trees, neural networks, and logistic regressions on parallel performance data. Additionally, we study noise in parallel performance data, whether the removal of noise and pre-processing of the data is necessary to accurately classify parallel sorting algorithms, and determine the effectiveness of features created from performance data. In conclusion, we demonstrate classification accuracy for these five different models of up to 97.7% across four different parallel algorithm classes.

Algorithm Classification↗

DOE OSTI2022

Spatial and temporal prediction of radiation dose rates near Fukushima Daiichi Nuclear Power Plant

In this paper, we have developed a methodology to estimate the spatiotemporal distribution of radiation air dose rates around the Fukushima Daiichi Nuclear Power Plant (FDNPP). In our exploratory data analysis, we found that (1) the temporal evolution of dose rates is composed of a log-linear decay trend and fluctuations of air dose rates that are spatially correlated among adjacent monitoring posts; and (2) the slope of the log-linear environmental decay trend can be represented as a function of the apparent initial dose rates, coordinate position, land-use type, and soil type. From these observations, we first estimated the log-linear decay trend at each location based on these predictors, using the random forest method. We then developed a modified Kalman filter coupled with a Gaussian process model to estimate the dose-rate time series at a given location and time. We applied this method to the Fukushima evacuation zone (as of March 2017), which included 17 monitoring post locations (with monitoring datasets collected between 2014 and 2018) and generated a time series of dose-rate maps. Our results show that this approach allows us to produce accurate spatial and temporal predictions of radiation dose-rate maps using limited spatiotemporal measurements.

61 RADIATION PROTECTION AND DOSIMETRY↗

DOE OSTI2024

Introducing Molecular Hypernetworks for Discovery in Multidimensional Metabolomics Data

Orthogonal separations of data from high-resolution mass spectrometry can provide insight into sample composition and address challenges of complete annotation of molecules in untargeted metabolomics. “Molecular networks” (MNs), as used in the Global Natural Products Social Molecular Networking platform, are a prominent strategy for exploring and visualizing molecular relationships and improving annotation. MNs are mathematical graphs showing the relationships between measured multidimensional data features. MNs also show promise for using network science algorithms to automatically identify targets for annotation candidates and to dereplicate features associated with a single molecular identity. Here, this paper introduces “molecular hypernetworks” (MHNs) as more complex MN models able to natively represent multiway relationships among observations. Compared to MNs, MHNs can more parsimoniously represent the inherent complexity present among groups of observations, initially supporting improved exploratory data analysis and visualization. MHNs also promise to increase confidence in annotation propagation, for both human and analytical processing. We first illustrate MHNs with simple examples, and build them from liquid chromatography- and ion mobility spectrometry-separated MS data. We then describe a method to construct MHNs directly from existing MNs as their “clique reconstructions”, demonstrating their utility by comparing examples of previously published graph-based MNs to their respective MHNs.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2020

Tensor factorization for elucidating mechanisms of piezoresponse relaxation via dynamic Piezoresponse Force Spectroscopy

Spatially resolved time and voltage-dependent polarization dynamics in PbTiO 3 thin films is explored using dynamic piezoresponse force microscopy (D-PFM) in conjunction with interferometric displacement sensing. This approach gives rise to 4D data sets containing information on bias-dependent relaxation dynamics at each spatial location without long-range electrostatic artifacts. To interpret these data sets in the absence of defined physical models, we employ a non-negative tensor factorization method which clearly presents the data as a product of simple behaviors allowing for direct physics interpretation. Correspondingly, we perform phase-field modeling finding the existence of ‘hard’ and ‘soft’ domain wall edges. This approach can be extended to other multidimensional spectroscopies for which even exploratory data analysis leads to unsatisfactory results due to many components in the decomposition.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Understanding GPU Memory Corruption at Extreme Scale: The Summit Case Study

GPU memory corruption and in particular double-bit errors (DBEs) remain one of the least understood aspects of HPC system reliability. Albeit rare, their occurrences always lead to job termination and can potentially cost thousands of node-hours, either from wasted computations or as the overhead from regular checkpointing needed to minimize the losses. As supercomputers and their components simultaneously grow in scale, density, failure rates, and environmental footprint, the efficiency of HPC operations becomes both an imperative and a challenge. We examine DBEs using system telemetry data and logs collected from the Summit supercomputer, equipped with 27,648 Tesla V100 GPUs with 2nd-generation high-bandwidth memory (HBM2). Using exploratory data analysis and statistical learning, we extract several insights about memory reliability in such GPUs. We find that GPUs with prior DBE occurrences are prone to experience them again due to otherwise harmless factors, correlate this phenomenon with GPU placement, and suggest manufacturing variability as a factor. On the general population of GPUs, we link DBEs to short- and long-term high power consumption modes while finding no significant correlation with higher temperatures. We also show that the workload type can be a factor in memory’s propensity to corruption.

Oles, Vlad↗

DOE OSTI2020

Machine Learning for Distributed Acoustic Sensing data (MLDAS) v1.0.1

MLDAS is a Python-written package for exploratory data analysis and deep learning training on Distributed Acoustic Sensing data. The machine learning tools are powered by the PyTorch library and designed to work efficiently on large scale datasets using parallel computing. Various SLURM scripts as well as a tutorial have also been made available to allow geophysicists to quickly and easily implement the available tools in their analysis workflow on supercomputer facilities.

Dumont, Vincent↗

DOE OSTI2023

LLNL/thicket

Thicket is a python-based toolkit for Exploratory Data Analysis (EDA) of parallel performance data that enables performance optimization and understanding of applications’ performance on supercomputers. It bridges the performance tool gap between being able to consider only a single instance of a simulation run (e.g., single platform, single measurement tool, or single scale) and finding actionable insights in multi-dimensional, multi-scale, multi-architecture, and multi-tool performance datasets.

Brink, Stephanie Labasan↗

DOE OSTI2023

Development of a hybrid neural network and transfer learning model for optimized ICP-MS/MS operation

Correct function and calibration of instrumentation is a crucial assumption for any scientific experiment. One such instrument, tandem inductively coupled plasma mass spectrometer (ICP-MS/MS), has in-depth calibration settings that range across 30+ different parameters, making it difficult to determine optimal conditions without expertise and some degree of trial and error. Often, these settings are hand-tuned, a time-intensive process prone to local maxima and human error. While some automation is available, the automation also may favor local optimizations over a global optimum. In addition to these difficulties, day to day instrument variability can further complicate the calibration process. We propose a solution to this problem as a machine learning (ML) algorithm that learns how each parameter helps determine the calibration sensitivity across several elements, and re-weights parameters over time as instrument variability changes (e.g., a global neural network (NN) with a time-dependent transfer learning (TL) component). This model would be able to generate a surface of predicted calibration sensitivities and their respective parameters, and a simple multivariate algorithm would be able to pull out the optimum results with the settings associated with them. Here-in, we describe our initial findings in working towards this goal, including data extraction from historical files, exploratory data analysis, and some initial model building to better describe the data and the feasibility of our goal.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2021

Exploratory analysis and performance prediction of big data transfer in High-performance Networks

Big data transfer in large-scale scientific and business applications is increasingly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) via advance bandwidth reservation. Provisioning agents need to carefully schedule data transfer requests, compute network paths, and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized, could be simply wasted due to the exclusive access during the approved time window, and cause extra overhead and complexity for resource management. This calls for accurate performance prediction to reserve bandwidths that match actual needs and avoid over-provisioning. We employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements collected in the past several years from data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated testbeds. We first analyze the performance patterns in response to a comprehensive list of parameters in end-host systems, network connections, and data transfer applications, which motivate the use of machine learning and also help us identify the effects of latent factors. We then propose threshold- and clustering-based methods to eliminate negative effects of latent factors in data preprocessing and build a robust performance predictor based on customized domain-oriented loss functions. The performance of the proposed methods is verified by extensive experiments using SVR and RFR as well as theoretical analysis of the general performance bound.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Physics-Informed Machine Learning for Epidemiological Models

One challenge of using compartmental SEIR models for public health planning is the difficulty in manually tuning parameters to capture behavior reflected in the real-world data. This team conducted initial, exploratory analysis of a novel technique to use physics-informed machine learning tools to rapidly develop data-driven models for physical systems. This machine learning approach may be used to perform data assimilation of compartment models which account for unknown interactions between geospatial domains (i.e. diffusion processes coupling across neighborhoods/counties/states/etc.). Results presented here are early, proof-of-concept ideas that demonstrate initial success in using a physically informed neural network (PINN) model to assimilate data in a compartmental epidemiology model. The results demonstrate initial success and warrant further research and development.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2021

Gaussian process analysis of electron energy loss spectroscopy data: multivariate reconstruction and kernel control

Abstract Advances in hyperspectral imaging including electron energy loss spectroscopy bring forth the challenges of exploratory and physics-based analysis of multidimensional data sets. The multivariate linear unmixing methods generally explore similarities in the energy dimension, but ignore correlations in the spatial domain. At the same time, Gaussian process (GP) explicitly incorporate spatial correlations in the form of kernel functions but is computationally intensive. Here, we implement a GP method operating on the full spatial domain and reduced representations in the energy domain. In this multivariate GP, the information between the components is shared via a common spatial kernel structure, while allowing for variability in the relative noise magnitude or image morphology. We explore the role of kernel constraints on the quality of the reconstruction, and suggest an approach for estimating them from the experimental data. We further show that spatial information contained in higher-order components can be reconstructed and spatially localized.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Measurement of the Neutron Electromagnetic Form Factor Ratio at High Momentum Transfer

The inner structure of the nucleon (proton and neutron) remains a topic of great interest in nuclear and particle physics, after many decades of study. For example, understanding the quark-gluon dynamics inside the nucleon would shed light on how 99% of the nucleon mass is created. The neutron electromagnetic form factors, Gn E and Gn M , give important insights into the neutron structure. The Super BigBite Spectrometer (SBS) program at Jefferson Lab (JLab) seeks to extend the form factor measurements for both the proton and the neutron. The neutron electric form actor, Gn E , has been historically difficult to measure due to the short lifetime of the free neutron and the small value of Gn E . The GEn-II experiment is part of the SBS program and seeks to measure Gn E , significantly increasing the high momentum transfer coverage. A newly designed polarized 3He target increased the figure of merit by three times compared to previous measurements. The analysis of this data is especially challenging due to the unprecedented high-rate environment caused by the open nature of the spectrometer with a direct line of sight to the target. This required developing new Gas Electron Multiplier (GEM) particle trackers which can cover large areas demanded by this setup and handle particle rates up to 500 kHz/cm2. Rates this high over a large area is unprecedented in particle tracking systems and came with a number of challenges. Data taken in the SBS program was critical to understanding hardware and software solutions that improved the track reconstruction efficiency to be >97% with a position resolution of 70 ?m. In previous experiments the proton electromagnetic form factors, Gp E and Gp M were measured up to Q2 = 8.5 GeV2 and Q2 = 30 GeV2, respectively, while Gn E has only been measured up to Q2 = 3.4 GeV2. The GEn-II experiment has measured the neutron form factor ratio, Gn E/Gn M, at Q2 values of 2.90, 6.50, and 9.47 GeV2 by scattering a polarized electron beam with a polarized 3He target, used here as an effective polarized neutron target, and measuring the double spin asymmetry of the cross section. Previous Gn E measurements do not extend above Q2 = 3.4 GeV2, and therefore this analysis has extended the world data by almost three times. The background correction is especially difficult at the higher Q2 settings leading to large systematic errors. As very exploratory results from this early analysis of the data, we find for Q2 = 2.90 GeV2, Gn E = 0.0157 ±stat 0.0016 ±sys 0.0011, for Q2 = 6.50 GeV2, Gn E = 0.0067 ±stat 0.0019 ±sys 0.0005, and for Q2 = 9.46 GeV2, Gn E = 0.0046 ±stat 0.0023 ±sys 0.0005. These results are compared to predictions from the Dyson-Schwinger Equations (DSE) model and a Relativistic Constituent Quark Model (RCQM).

Jeffas, Sean↗

DOE OSTI2024

Data and scripts associated with a manuscript on a meta-analysis synthesizing stream biogeochemical response to wildfires across space and time (v2)

This data package is associated with the publication “Catchment characteristics modulate the influence of wildfires on nitrate and dissolved organic carbon in lotic systems across space and time: A meta-analysis” submitted to Global Biogeochemical Cycles (Cavaiani et al. 2025). This study uses meta-analytical techniques to evaluate the effect of wildfire on in-stream responses in burned and unburned watersheds. The study aims to provide additional insight into the range of responses and net influences that wildfires have on hydro-biogeochemistry across broad spatial scales, burn extents, and the persistence of water-quality change. This study compiles data and metadata from 18 total publications that includes 1) surface water geochemistry data (dissolved organic carbon; nitrate), 2) climate classifications, 3) year of the wildfire, 4) the time lag between when the fire occurred and when the sampling occurred, and 5) study design of the publication. In total, this meta-analysis draws data that spans 8 climate guilds, 3 biomes, 62 watersheds, and 20 unique wildfires. See Sites_meta_data.csv for citations of the papers used in this meta-analysis. All R scripts and the associated data can also be found on GitHub at This data package was originally published in March 2024. It was updated in April 2025 (v2; new and modified files). See the change history section in the readme for more details. This data package contains five primary folders that include the following: (1) inputs; (2) output for analysis; (3) initial plots; (4) R scripts; and (5) GIS data. The data package also contains a data dictionary (dd) that provides column header definitions and a file-level metadata (flmd) file that describes every file. The “inputs” folder contains a list of all publications identified during the formal web search and an indication of whether each publication was included in the final analysis. Additionally, it includes site-level metadata, catchment characteristics, and GIS data for all publications included in the final analysis. The “Output_for_analysis” folder contains all data frames and figures generated from each R script used for additional data analysis. The “initial_plots” folder includes all exploratory figures that will be included in a supplemental and figures that will be submitted with the manuscript for publication. The “R_scripts” folder contains the scripts that perform all the data manipulations, statistical analyses, and plots. The “gis_data” folder includes shape files for each fire included in this meta-analysis. This data package contains the following file types: csv, pdf, jpeg, cpg, dbf, prj, shp, shp.ea.iso.xml, shp.iso.xml, shx.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2020

A simple and fast algorithm for estimating the capacity credit of solar and storage

Energy storage is a leading option to enhance the resource adequacy contribution of solar energy. Detailed analysis of the capacity credit of solar energy and energy storage is limited in part due to the data intensive and computationally complex nature of probabilistic resource adequacy assessments. This paper presents a simple algorithm for calculating the capacity credit of energy-limited resources that, due to the low computational and data needs, is well suited to exploratory analysis. Validation against benchmarks based on probabilistic techniques shows that it can yield similar insights. The method is used to evaluate the impact of different solar and storage configurations, particularly with respect to the strategy for coupling storage and solar photovoltaic systems. Furthermore, application of the method to a case study of utilities in Florida, where solar is rapidly growing and demand peaks in the winter and summer, demonstrates that it can improve on rules of thumb used in practice by some utilities. If storage is required to charge only from solar, periods of high demand driven by cold weather events accompanied by lower solar production can result in a capacity credit of solar and storage that is less than the capacity credit of storage alone.

14 SOLAR ENERGY↗

DOE OSTI2020

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

DOE OSTI2020

Slycat Enables Synchronized 3D Comparison of Surface Mesh Ensembles [Brief]

In support of analyst requests for Mobile Guardian Transport studies, researchers at Sandia National Laboratories have expanded data types for the Slycat ensemble-analysis and visualization tool to include 3D surface meshes. This new capability represents a significant advance in our ability to perform detailed comparative analysis of simulation results. Analyzing mesh data rather than images provides greater flexibility for post-processing exploratory analysis.

36 MATERIALS SCIENCE↗

DOE OSTI2020

Theory and applications of parton pseudodistributions

We review the basic theory of the parton pseudodistributions approach and its applications to lattice extractions of parton distribution functions. The crucial idea of the approach is the realization that the correlator $M(z,p)$ of the parton fields is a function $M(ν,-z^2)$ of Lorentz invariants $ν=-(zp)$, the Ioffe time, and the invariant interval $z^2$. This observation allows to extract the Ioffe-time distribution $M(ν,-z^2)$ from Euclidean separations z accessible on the lattice. Another basic feature is the use of the ratio $M(ν,-z^2)≡M(ν,-z^2)/M(0,-z^2)$, that allows to eliminate artificial ultraviolet divergence generated by the gauge link for spacelike intervals. The remaining $z^2$-dependence of the reduced Ioffe-time distribution $M(ν,-z^2)$ corresponds to perturbative evolution, and can be converted into the scale-dependence of parton distributions $f(x,μ^2)$ using matching relations. The ν-dependence of $M(ν,-z^2)$ governs the x-dependence of parton densities $f(x,μ^2)$. The perturbative evolution was successfully observed in exploratory quenched lattice calculation. The analysis of its precise data provides a framework for extraction of parton densities using the pseudodistributions approach. It was used in the recently performed calculations of the nucleon and pion valence quark distributions. We also discuss matching conditions for the pion distribution amplitude and generalized parton distributions, the lattice studies of which are now in progress.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗