Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “software: data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Coherent diffraction imaging in the undergraduate laboratory

We present an undergraduate optics instructional laboratory designed to teach skills relevant to a broad range of modern scientific and technical careers. In this laboratory project, students image a custom aperture using coherent diffraction imaging, while learning principles and skills related to digital image processing and computational imaging, including multidimensional Fourier analysis, iterative phase retrieval, noise reduction, finite dynamic range, and sampling considerations. After briefly reviewing these imaging principles, we describe the required experimental materials and setup for this project. Our experimental apparatus is both inexpensive and portable, and a software application we developed for interactive data analysis is freely available.

Porter, J. Nicholas↗
DOE OSTI2024

Ring Pull Strain Analysis

Ring Pull Strain Analysis (RPSA) is a framework for analysis of Digital Image Correlation (DIC) data of gaugeless ring pull testing. It is meant for analysis of rings cut from tube cross sections and pulled in a displacement-controlled tensile test. This code takes load frame data, images take during testing, and csv files from the DIC analysis of those images and allows the user to analyze and plot data from the test. Some important features of the software include getting the strain values as a function of radial distance and angle, plotting of the strain around the ring, and stress-strain analysis of the load frame data. Additional capability is included with Ring Pull Coating Analysis (RPCA, also part of the software). Additional images are taken of samples with an outer coating and are used to correlate failure of the coating with strain values measured with DIC.

Beck, Peter↗
DOE OSTI2021

Scalable, In-situ Data Clustering Data Analysis for Extreme Scale Scientific Computing (Final Report)

The objective of this project is to address challenges in the design and development of scalable in-situ data clustering and analytics algorithms and software. Our goal is to develop parallel software consisting of a set of spatio-temporal data clustering and anomaly detection functions, both of which are very important for large-scale analysis and have wide applicability for in-situ runs as well as post-processing analysis. Our design principles for in-situ analysis consider the following: (1) identify parts of the computation can be done close to the data within the nodes, while it is still in memory; (2) extract analysis components can (and should) be performed in remote staging and analysis nodes; (3) develop error-bound approximation methods for applications tolerable for small errors; (4) identify the type of derived distributions and statistics, for spatio-temporal data, that can be kept locally in order to both accelerate computations and meet energy constraints in subsequent iterations and phases; (5) use a self-describing data format so that data can be consistent and understood among local storage (memory and SSDs) and at staging and analysis nodes, thereby providing portability and flexibility; (6) develop service-oriented functions that can schedule in-situ and post-hoc analysis tasks based on the dynamic requirements of applications. Our development focus is to produce the parallel data analysis software/library that will be scalable, reusable, extensible, and generic for applications in different disciplines. The software will be able to run in-situ with the simulations as well as post-hoc analysis. This approach will satisfy many synergistic requirements for data intensive applications executed on data coming from instruments and experiments. In particular, the proposed multilevel approach is directly applicable to perform design tradeoffs for running part of the algorithms near the instruments and the rest on remote (analysis) systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Mass Spectrometer Transient Analysis

This software implements a complete preprocessing pipeline for transient mass spectrometry (MS) data collected during TAP (Temporal Analysis of Products) experiments. It is designed to extract chemically meaningful fluxes from overlapping ion signals by applying a calibrated defragmentation matrix and solving the resulting linear system using non-negative least squares (NNLS) regression. The core script, preprocess_mass_spec.py, performs the following operations: Gain correction: Applies amplifier gain scalars derived from inert-packed calibration pulses to normalize signal intensities across AMUs and acquisition settings. Background subtraction: Removes experiment baselines using user-defined time windows, ensuring compatibility with slow-diffusing species and preventing negative values that would interfere with NNLS. Options to subtract before and after defragmentation. Defragmentation: Constructs a fragmentation matrix A from zeroth moments of calibration pulses (equal molar gas:inert mixtures) and solves Ax=b at each time point, where b is the raw MS signal and x is the estimated species flux. The matrix is normalized to inert signals and accounts for instrument-specific fragmentation behavior. Pulse-mode handling: Supports both averaged and individual pulse modes, enabling statistical treatment of fluxes and calculation of standard deviations. Integration and output: Computes zeroth moments (integrated fluxes) and exports time-resolved and integrated data in CSV format, suitable for downstream kinetic modeling. The software is validated using both virtual TAP simulations (VTAP) and experimental data from propane dehydrogenation (PDH) on CrOx/Al2O3 catalysts. It preserves temporal resolution by applying NNLS point-by-point across the pulse duration (typically 6,000+ time slices per pulse), leveraging the linear superposition principle to reconstruct full flux profiles. The defragmented outputs are compatible with kinetic extraction methods such as the G and Y procedures, which are used to derive rate–concentration relationships from TAP data. The details of these validations are discussed in detail in the supporting manuscript and supporting information. Example data and output files are also included. The methodology is robust to experimental noise and drift, with calibration protocols that account for pulse size effects, MS aging, and inert gas normalization. The software is modular, reproducible, and tailored for high-throughput TAP-MS workflows in catalysis research.

Kristy, Stephen [Idaho National Laboratory (INL), ↗
DOE OSTI2024

CHMMPP: A c++ library for constrained Hidden Markov Models

SAND2024-13027O The CHMMPP: A c++ Library for Constrained Hidden Markov Models (HMM) software supports the analysis of multivariate time series data to detect patterns using HMM. Many applications involve the detection and characterization of hidden or latent states in a complex system using observable states and variables. This software supports inference of latent states integrating both an HMM and application-specific constraints that reflect known relationships in hidden states. The CHMMPP software supports application-specific and generic methods for constrained inference. This includes a framework for customized Viterbi methods, constrained inference of hidden states with A* and integer programming methods, and various constraint-informed methods for learning HMM model parameters. CHMMPP focuses on supporting generic methods that enable the agile expression of complex sets of constraints that naturally arise in many real-world applications.

Hart, William↗
DOE OSTI2026

Coupling of high-resolution mass spectrometer and photosynthesis system for comprehensive leaf volatile metabolite profiling

Background Leaf-level biogenic volatile organic compounds (BVOCs) emissions represent a major source of organic gases in the atmosphere, influencing both climate and air quality. These emissions are strongly driven by environmental perturbations, which affect individual plant- to ecosystem-level processes. Uncovering all the BVOCs and understanding how their emissions respond to altered environmental conditions provide critical insights into vegetation-driven changes in atmospheric chemistry. We developed a tandem instrumentation setup that integrates a proton transfer reaction time-of-flight mass spectrometer (PTR-ToF-MS) with parts-per-trillion detection limits and a photosynthetic infrared gas exchange system for the untargeted survey of all the BVOCs. This novel system enables simultaneous, real-time monitoring of BVOC emissions and photosynthetic parameters at the leaf level, offering new opportunities to disentangle the physiological and environmental drivers of VOC release. Furthermore, we established the VOC Analysis and Processing Optimization Resource (VAPOR), an open-access software tool designed for rapid data post-processing and the analysis of the variability of hundreds of BVOCs. We assessed the performance of the tandem system under varying background conditions, using standard gas mixtures and a range of environmental factors. Results Blank emissions were substantially lower for major BVOCs (e.g., isoprene) compared to those observed in plant emissions. Despite this, the observation of background-level VOCs highlights the importance of routinely acquiring and accounting for blank measurements in analyses using the coupled instrumentation. Introduction of known VOC concentrations to the system demonstrated a linear response across different compounds with varying molecular compositions, indicating minimal gas loss regardless of chemical moieties within the coupled instrumentation. We applied the optimized system to investigate the physiological mechanisms driving BVOC emissions across different genotypes of poplar and pennycress. The high mass resolution capabilities of the PTR-ToF-MS, coupled with comprehensive VAPOR-driven data analysis, enabled the identification of several important BVOCs, including methanol and methanethiol; these BVOCs displayed substantial variation across pennycress genotypes and showed concentrations ~ 100–350% higher than the blank. Moreover, isoprene emissions varied significantly among poplar genotypes grown in different potting media. Conclusions Tandem instrumentation offers a powerful tool for profiling volatile molecular markers and elucidating their genetic and environmental underpinnings. This approach enhances our ability to predict BVOC emissions in response to genotype by environmental interactions and contributes to a deeper understanding of vegetation responses to environmental changes.

Biogenic volatile organic compounds↗
DOE OSTI2023

Building an Efficient Cluster Cosmology Software Package for Modeling Cluster Counts and Lensing

We introduce a software suite developed for galaxy cluster cosmological analysis with the Dark Energy Survey Data. Cosmological analyses based on galaxy cluster number counts and weak-lensing measurements need efficient software infrastructure to explore an increasingly large parameter space, and account for various cosmological and astrophysical effects. Our software package is designed to model the cluster observables in a wide-field optical survey, including galaxy cluster counts, their averaged weak-lensing masses, or the cluster's averaged weak-lensing radial signals. To ensure maximum efficiency, this software package is developed in C++ in the CosmoSIS software framework, making use of the CUBA integration library. We also implement a testing and validation scheme to ensure the quality of the package. We demonstrate the effectiveness of this development by applying the software to the Dark Energy Survey Year 1 galaxy cluster cosmological data sets, and acquired cosmological constraints that are consistent with the fiducial Dark Energy Survey analysis.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

Generic and ML Workloads in an HPC Datacenter: Node Energy, Job Failures, and Node-Job Analysis

HPC datacenters offer a backbone to the modern digital society. Increasingly, they run Machine Learning (ML) jobs next to generic, compute-intensive workloads, supporting science, business, and other decision-making processes. However, understanding how ML jobs impact the operation of HPC datacenters, relative to generic jobs, remains desirable but understudied. In this work, we leverage long-term operational data, collected from a national-scale production HPC datacenter, and statistically compare how ML and generic jobs can impact the performance, failures, resource utilization, and energy consumption of HPC datacenters. Our study provides key insights, e.g., ML-related power usage causes GPU nodes to run into temperature limitations, median/mean runtime and failure rates are higher for ML jobs than for generic jobs, both ML and generic jobs exhibit highly variable arrival processes and resource demands, significant amounts of energy are spent on unsuccessfully terminating jobs, and concurrent jobs tend to terminate in the same state. We open-source our cleaned-up data traces on Zenodo (https://doi. org/10.5281/zenodo.13685426), and provide our analysis toolkit as software hosted on GitHub (https://github.com/atlarge-research/2024-icpads-hpc-workload-characterization). This study offers multiple benefits for data center administrators, who can improve operational efficiency, and for researchers, who can further improve system designs, scheduling techniques, etc.

crossanalysis↗
DOE OSTI2024

10 Boron-film-based gas detectors at ESS

The development of detectors for the European Spallation Source is an important parallel element to the effort put into the design and construction of the neutron source and instruments. The 10Boron-film-based detector developments that started over a decade ago as basic detector concepts have now reached technical maturity after intense prototyping work and numerous testing campaigns. Several of the ESS beamlines that will soon enter the commissioning phase started welcoming the detector systems built with the 10 B-film converter technology. The real-size demonstrators evolved in the last 2–3 years into a diverse suite of gas proportional counters for use in diffraction, reflectometry and small-angle scattering studies in spite of the operational complexity posed by the requirements for large-area coverage, low material budget and robustness for operation in the high-flux and high-radiation environment of ESS. The detectors are now being shipped to ESS or undergoing the last performance and calibration tests before being handed over to the instrument teams for installation in the host beamlines. A common feature of these detector systems is the very large number of readout channels that they are instrumented with in order to fulfill the demanding requirements for sensitivity, spatial resolution and count-rate capability. Across all of the detector types that will operate at ESS, the integration, testing and commissioning of the read-out technologies and software tools for data reduction, calibration and analysis are the focus of the detector and integration teams. These will yield a wealth of knowledge about their operation as well as initial results on the in-situ performance, a very important asset to ensure rapid commissioning of the detectors when neutrons from the ESS source become available. In this paper we will give an overview of the 10 B-film-based detector technologies included in the ESS detector suite that are currently facing the transition from the production phase to installation and integration with the other beamline components, which comes with its own specific challenges of both organizational and technical nature.

47 OTHER INSTRUMENTATION↗
DOE OSTI2023

BatAnalysis - A Comprehensive Python Pipeline for Swift BAT Survey Analysis

The Swift Burst Alert Telescope (BAT) is a coded-aperture gamma-ray instrument with a large field of view that primarily operates in survey mode when it is not triggering on transient events. The survey data consist of 80-channel detector plane histograms that accumulate photon counts over periods of at least 5 minutes. These histograms are processed on the ground and are used to produce the survey data set between 14 and 195 keV. Survey data comprise >90% of all BAT data by volume and allow for the tracking of long-term light curves and spectral properties of cataloged and uncataloged hard X-ray sources. Until now, the survey data set has not been used to its full potential due to the complexity associated with its analysis and the lack of easily usable pipelines. Here, we introduce the BatAnalysis Python package, a wrapper for HEASoftpy, which provides a modern, open-source pipeline to process and analyze BAT survey data. BatAnalysis allows members of the community to use BAT survey data in more advanced analyses of astrophysical sources, including pulsars, pulsar wind nebula, active galactic nuclei, and other known/unknown transient events that may be detected in the hard X-ray band. We outline the steps taken by the Python code and exemplify its usefulness and accuracy by analyzing survey data of the Crab Nebula, NGC 2992, and a previously uncataloged MAXI transient. The BatAnalysis package allows for ~18 yr of BAT survey data to be used in a systematic way to study a large variety of astrophysical sources.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Earth and Space Science Informatics Perspectives on Integrated, Coordinated, Open, Networked (ICON) Science

Abstract This article is composed of three independent commentaries about the state of Integrated, Coordinated, Open, Networked (ICON) principles (Goldman, et al., 2021b, https://doi.org/10.1029/2021EO153180 ) in Earth and Space Science Informatics (ESSI) and includes discussion on the opportunities and challenges of adopting them. Each commentary focuses on a different topic: (Section 2) Global collaboration, cyberinfrastructure, and data sharing; (Section 3) Machine learning for multiscale modeling; (Section 4) Aerial and satellite remote sensing for advancing Earth system model development by integrating field and ancillary data. ESSI addresses data management practices, computation and analysis, and hardware and software infrastructure. Our role in ICON science therefore involves collaborative work to assess, design, implement, and promote practices and tools that enable effective data management, discovery, integration, and reuse for interdisciplinary work in Earth and space science disciplines. Networks of diverse people with expertise across Earth, space, and data science disciplines are essential for efficient and ethical exchanges of findable, accessible, interoperable, and reusable (FAIR) research products and practices. Our challenge is then to coordinate the development of standards, curation practices, and tools that enable integrating and reusing multiple data types, software, multi‐scale models, and machine learning approaches across disciplines in a way that is as open and/or FAIR as ethically possible. This is a major endeavor that could greatly increase the pace and potential of interdisciplinary scientific discovery.

58 GEOSCIENCES↗
DOE OSTI2022

Implementation Aspects of Smart Grids Cyber-Security Cross-Layered Framework for Critical Infrastructure Operation

Communication networks in power systems are a major part of the smart grid paradigm. It enables and facilitates the automation of power grid operation as well as self-healing in contingencies. Such dependencies on communication networks, though, create a roam for cyber-threats. An adversary can launch an attack on the communication network, which in turn reflects on power grid operation. Attacks could be in the form of false data injection into system measurements, flooding the communication channels with unnecessary data, or intercepting messages. Using machine learning-based processing on data gathered from communication networks and the power grid is a promising solution for detecting cyber threats. In this paper, a co-simulation of cyber-security for cross-layer strategy is presented. The advantage of such a framework is the augmentation of valuable data that enhances the detection as well as identification of anomalies in the operation of the power grid. The framework is implemented on the IEEE 118-bus system. The system is constructed in Mininet to simulate a communication network and obtain data for analysis. A distributed three controller software-defined networking (SDN) framework is proposed that utilizes the Open Network Operating System (ONOS) cluster. According to the findings of our suggested architecture, it outperforms a single SDN controller framework by a factor of more than ten times the throughput. This provides for a higher flow of data throughout the network while decreasing congestion caused by a single controller’s processing restrictions. Furthermore, our CECD-AS approach outperforms state-of-the-art physics and machine learning-based techniques in terms of attack classification. The performance of the framework is investigated under various types of communication attacks.

cross-layered↗
DOE OSTI2024

Real‐time XFEL data analysis at SLAC and NERSC: A trial run of nascent exascale experimental data analysis

X‐ray scattering experiments using free electron lasers (XFELs) are a powerful tool to determine the molecular structure and function of unknown samples (such as COVID‐19 viral proteins). XFEL experiments are a challenge to computing in two ways: (i) due to the high cost of running XFELs, a fast turnaround time from data acquisition to data analysis is essential to make informed decisions on experimental protocols; (ii) data‐collection rates are growing exponentially, requiring new scalable algorithms. Here we report our experiences analyzing data from two experiments at the Linac Coherent Light Source (LCLS) during September 2020. Raw data were analyzed on NERSC's Cori XC40 system, using the Superfacility paradigm: our workflow automatically moves raw data between LCLS and NERSC, where it is analyzed using the software package CCTBX. We achieved real time data analysis with a turnaround time from data acquisition to full molecular reconstruction in as little as 10 min—sufficient time for the experiment's operators to make informed decisions. By hosting the data analysis on Cori, and by automating LCLS‐NERSC interoperability, we achieved a data analysis rate which matches the data acquisition rate. Completing data analysis within 10 min is a first for XFEL experiments and an important milestone if we are to keep up with data‐collection trends.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2021

Real-Time XFEL Data Analysis at SLAC and NERSC: a Trial Run of Nascent Exascale Experimental Data Analysis

X-ray scattering experiments using Free Electron Lasers (XFELs) are a powerful tool to determine the molecular structure and function of unknown samples (such as COVID-19 viral proteins). XFEL experiments are a challenge to computing in two ways: i) due to the high cost of running XFELs, a fast turnaround time from data acquisition to data analysis is essential to make informed decisions on experimental protocols; ii) data collection rates are growing exponentially, requiring new scalable algorithms. Here we report our experiences analyzing data from two experiments at the Linac Coherent Light Source (LCLS) during September 2020. Raw data were analyzed on NERSC's Cori XC40 system, using the Superfacility paradigm: our workflow automatically moves raw data between LCLS and NERSC, where it is analyzed using the software package CCTBX. We achieved real time data analysis with a turnaround time from data acquisition to full molecular reconstruction in as little as 10 min -- sufficient time for the experiment's operators to make informed decisions. By hosting the data analysis on Cori, and by automating LCLS-NERSC interoperability, we achieved a data analysis rate which matches the data acquisition rate. Furthermore, completing data analysis with 10 mins is a first for XFEL experiments and an important milestone if we are to keep up with data collection trends.

Blaschke, Johannes P.↗
DOE OSTI2021

Development of a Unified Taxonomy for HVAC System Faults

Detecting and diagnosing HVAC faults is critical for maintaining building operation performance, reducing energy waste, and ensuring indoor comfort. An increasing deployment of commercial fault detection and diagnostics (FDD) software tools in commercial buildings in the past decade has significantly increased buildings’ operational reliability and reduced energy consumption. A massive amount of data has been generated by the FDD software tools. However, efficiently utilizing FDD data for ‘big data’ analytics, algorithm improvement, and other data-driven applications is challenging because the format and naming conventions of those data are very customized, unstructured, and hard to interpret. This paper presents the development of a unified taxonomy for HVAC faults. A taxonomy is an orderly classification of HVAC faults according to their characteristics and causal relations. The taxonomy includes fault categorization, physical hierarchy, fault library, relation model, and naming/tagging scheme. The taxonomy employs both a physical hierarchy of HVAC equipment and a cause-effect relationship model to reveal the root causes of faults in HVAC systems. A structured and standardized vocabulary library is developed to increase data representability and interpretability. The developed fault taxonomy can be used for HVAC system ‘big data’ analytics such as HVAC system fault prevalence analysis or the development of an HVAC FDD software standard. A common type of HVAC equipment-packaged rooftop unit (RTU) is used as an example to demonstrate the application of the developed fault taxonomy. Two RTU FDD software tools are used to show that after mapping FDD data according to the taxonomy, the meta-analysis of the multiple FDD reports is possible and efficient.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2020

Application of Gaussian Bayes classifier to differentiate chlorine-based chemical agents

The Portable Isotopic Neutron Spectroscopy (PINS) is a commercialized system developed by Idaho National Laboratory (INL) to examine chemical warfare agents (CWA) non-destructively, utilizing Prompt Gamma Neutron Activation Analysis (PGNAA) techniques. The PINS system takes advantage of a high-resolution gamma-ray spectrum from a mechanically-cooled high-purity germanium (HPGe) detector. One of the difficult technical challenges is to discriminate the chlorine-based chemical agents. Especially, CN, CNB, CNS and CG have similar chemical compositions to make it hard to discriminate them with a higher confidence. Current identification algorithms for PINS systems with 252-Cf sources have been improved and updated continuously as more field data became available, and new algorithms was studied to complement the current algorithms by adopting the Gaussian Bayes classifier. These new algorithms were intended to be applied to a subset of chlorine-based chemical agents, and their main goal is discriminate CN, CNB, CNS and CG with their ratios of the chlorine neutron inelastic 1763keV peak to the chlorine thermal neutron capture 1959keV peak, which is referred to as the “Cl i/c” or “clic” ratio in this study. The Cl i/c ratios were assumed to follow Gaussian distributions with the means and the standard deviations unique to their corresponding chemical agents. The prior probabilities of these four chemical agents were optimized with a collection of field data to achieve the best performance in terms of precision or positive predictive value (PPV). Finally, their posterior probabilities as functions of the Cl i/c ratio were implemented in the current version of PINS analysis software in order to be tested with more field data.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗
DOE OSTI2021

Challenges and Solutions for Real-Time Phasor Modeling of Large-scale Distribution Network with High PV Penetration

The conversion process of a practical large-scale feeder data from a quasi-static time series (OpenDSS) model into a real-time phasor model (ePHASORSIM in Opal-RT) is discussed in the paper. The process is implemented using an open source Python software. Previous reported implementations for the conversion process lead to several errors when applied to a larger-scale system such as the one considered here. Hence in this work, we describe the common issues in this conversion and propose a customized solution to enhance the efficiency of the conversion and reduce the complexity in the process. A quantitative validation of the enhanced conversion process is presented in this work using an actual high PV penetration feeder model that consists of 2230 buses, and using actual load and PV profile data. After a detailed analysis, this customized conversion software will be made available as an open source tool and is expected to be helpful for researchers who want to pursue a similar conversion. Solutions to various observed issues such as identifying the lines due to islanded network, representation of full impedance model of transformer/lines as sequential models, complexity in the representation of single phase buses/lines as three phase buses/lines to make it compatible with the simulator platform are discussed. Comparison of power flow and time series simulation results obtained from both OpenDSS and ePHASORsim models show very low errors, validating the accuracy of the proposed conversion process.

14 SOLAR ENERGY↗
DOE OSTI2022

PVAnalytics: A Python Package for Automated Processing of Solar Time Series Data

Multiple publicly available software packages exist that analyze solar time series data, including RdTools and Solar Data Tools, among others. Several of these packages contain their own unique quality assurance (QA) and feature recognition algorithms. The python PVAnalytics package was developed to offer an internally consistent source for these analysis tools, making it easier for the end user to deploy these routines on his or her solar data. The PVAnalytics package currently contains routines for outlier detection, inverter clipping detection, irradiance and temperature checks, orientation checks, and data shift detection, among other functions. These functions have been aggregated from various sources including Solar Forecast Arbiter, RdTools, and the QA process developed by NREL's PV Fleets Initiative. We are continuously adding new functionality to the package, including documentation, examples and algorithms. By bundling QA functionality into a single software package, we hope to make PVAnalytics a comprehensive software library to support analysis of solar metadata and time series data.

data cleaning↗