Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “automated preprocessing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS1982

Automated preprocessing of spaceborne SAR data

An efficient algorithm has been developed for estimation of the echo phase delay in spaceborne synthetic aperture radar (SAR) data. This algorithm utilizes the spacecraft ephemeris data and the radar echo data to produce estimates of two parameters: (1) the centroid of the Doppler frequency spectrum f(d) and (2) the Doppler frequency rate. Results are presented from tests conducted with Seasat SAR data. The test data indicates that estimation accuracies of 3 Hz for f(d) and 0.3 Hz/sec for the Doppler frequency rate are attainable. The clutterlock and autofocus techniques used for estimation of f(d) and the Doppler frequency rate, respectively are discussed and the algorithm developed for optimal implementation of these techniques is presented.

Curlander, J. C.↗
DOE OSTI2021

Removal of spurious data in Bragg coherent diffraction imaging: an algorithm for automated data preprocessing

Bragg coherent diffraction imaging (BCDI) provides a powerful tool for obtaining high-resolution structural information from nanocrystalline materials. Here a BCDI sample consisting of a large number of randomly oriented nanoscale crystals is considered. Ideally, only one crystal is oriented to produce a Bragg peak on the detector. However, diffraction from other crystals often produces additional signals on the detector. Before the measured diffraction patterns can be processed into structural images, scientists routinely need to manually identify and remove the `alien' intensities from sources other than the intended crystal. With the development of modern high-coherence storage rings, such as the upgraded Advanced Photon Source (APS), the already slow process of manual preprocessing will be untenable for the large volumes of data that will be produced. An automated method of identifying and deleting alien intensities is proposed. This method exploits the fact that BCDI of a perfect crystal produces diffraction data with inversion symmetry around the Bragg peak. This approach uses the machine learning clustering method DBSCAN to distinguish between diffraction from multiple sources, and then calculates cluster size and inversion symmetry to assess whether clusters of intensity belong to desired data or alien signals. This approach can dramatically reduce the amount of time spent manually processing data, allowing BCDI data processing capabilities to keep pace with the technological advances of fourth-generation synchrotron light sources.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Creating a Training Dataset for Semantic Segmentation of Canal Networks for Irrigation Modernization

Canal infrastructure has provided critical irrigation water to the western United States for over a century. To continue providing vital water resources to the semi-arid West, irrigation systems must undergo maintenance and modernization. Many canal companies are resource-constrained, and because funding opportunities often require detailed knowledge of existing infrastructure, they can struggle to secure financial capital. We address this problem by creating training data for a semantic segmentation deep learning model to map canal networks throughout the western United States. To create a diverse and robust training dataset, we labelled 1-m NAIP imagery with the locations of no canals, wet canals, and dry/vegetated canals. Since creating these datasets is time consuming, we first developed a preprocessing methodology to identify canals within our four study areas. We used NAIP imagery and provided canal centerline data to buffer, standardize, and cluster the imagery, automating the labeling process as much as possible. However, this still required manual cleaning and manual classification of canal type. Challenges arose when canals were interrupted (e.g., road culverts or piped sections) or when nearby features shared similar characteristics (e.g., irrigated fields, trees, and shadows). Combining automated preprocessing with manual refinement produced four detailed canal masks to be used in the semantic segmentation model developed by Richard Tapia.

13 - HYDRO ENERGY↗
NASA NTRSDate not supplied

Predicting Team Functioning in Long Term Space Missions Using Acoustic and Linguistic Measures

Maintaining optimal team functioning is critical for long-duration space exploration missions, yet traditional monitoring methods, such as self-reports and wearable sensors, often impose operational burdens or suffer from bias. This paper investigates a non-intrusive speech-based artificial intelligence (AI) framework to predict degradations in team functioning using data from the Human Exploration Research Analog (HERA) of the U.S. National Aeronautics and Space Administration (NASA). Using acoustic features, linguistic descriptors, and semantic embeddings, we evaluate static non-linear and temporal machine learning models to predict both objective (task accuracy) and subjective (self-reported efficacy and cohesion) team functioning outcomes. Results indicate that temporal models outperform static approaches, with prediction of objective task accuracy in Team Interaction Battery (TIB) improving from near chance to 71%. Self-reported outcomes, including team efficacy and cohesion, are predicted more reliably than task performance, achieving balanced accuracies of up to 85.56% and 78.12%, respectively, and are found to be most strongly associated with acoustic features. In a second interdependent task, the MMSEV–EVA, accuracies of up to 78% are achieved using temporal models with acoustic features. Furthermore, incorporating just 1–2 days of team-specific historical data systematically improved performance, and acoustic markers from informal pre-task interactions provided modest predictive gains. Finally, while automated preprocessing yielded viable accuracy, humancorrected data provided moderate performance gains, though transcription error rates did not significantly correlate with model performance. These findings highlight the potential of speech as a passive, high-fidelity monitoring tool for autonomous habitats.

Temporal modeling↗
DOE OSTI2025

S AP F LOWER : an automated tool for sap flow data preprocessing, gap-filling, and analysis using deep learning

Sap flow, a critical process in plant water use and ecosystem water cycles, is often measured using thermal dissipation probes (TDP) due to their ease of installation and continuous data collection. However, sap flow data frequently include noise, outliers, and gaps, creating challenges for analysis and requiring substantial manual processing. We developed S AP F LOWER , a tool that automates data preprocessing, model training, gap-filling, sapwood area scaling and modeling, and water use analysis. It integrates autocleaning, machine learning and deep learning models (e.g. random forest, Gaussian process regression, long short-term memory (LSTM), bidirectional LSTM (BiLSTM)), and efficient workflows to process sap flow data. S AP F LOWER can remove over 90% of noisy data while preserving legitimate variations and achieve high accuracy in gap-filling based on user-determined parameters. Random forest, LSTM, and BiLSTM models reduced root mean square error to 10% or less for long-term gaps. Model training and prediction can be performed efficiently within seconds. S AP F LOWER significantly enhances the efficiency and accessibility of TDP data analysis by automating complex tasks, enabling researchers without programming expertise to employ advanced techniques. Future improvements will focus on species-specific corrections for TDP and support for additional measurement methods. S AP F LOWER is openly available on GitHub (https://github.com/JiaxinWang123/SapFlower) and Zenodo (doi: 10.5281/zenodo.13665919).

ecosystem water balance↗
NASA NTRS1996

ACTS (Advanced Communications Technology Satellite) Propagation Experiment: Preprocessing Software User's Manual

The preprocessing software manual describes the Actspp program originally developed to observe and diagnose Advanced Communications Technology Satellite (ACTS) propagation terminal/receiver problems. However, it has been quite useful for automating the preprocessing functions needed to convert the terminal output to useful attenuation estimates. Prior to having data acceptable for archival functions, the individual receiver system must be calibrated and the power level shifts caused by ranging tone modulation must be received. Actspp provides three output files: the daylog, the diurnal coefficient file, and the file that contains calibration information.

Crane, Robert K.↗
DOE OSTI2023

Survey of Time Shift Detection Algorithms for Measured PV Data

In this research, three variations of time shift detection algorithms were tested for their ability to detect time shift issues (including daylight savings time and random time shifts) in measured PV data sets. Two algorithms from the Python PVAnalytics package were assessed, and one algorithm from the Solar-Data-Tools package was assessed. Each algorithm's ability to accurately detect and measure time shifts was assessed.

automated preprocessing↗
NASA NTRS1985

Analysis of Spiral Bevel Gearing

Program generalized for automated finite-element-method model generation. Tedious task of finite-element-method (FEM) model preparation greatly reduced by development of preprocessing program for automated model generation. Utility of program, including ability to rapidly evaluate parametric changes, demonstrated for spiral, helical, and spiral bevel gearing. Postprocessing program also developed, aids in review and interpretation of calculated FEM stresses.

Drago, R. J.↗
DOE OSTI2024

RAP: Resource-aware Automated GPU Sharing for Multi-GPU Recommendation Model Training and Input Preprocessing

Ensuring high-quality recommendations for newly onboarded users requires the continuous retraining of Deep Learning Recommendation Models (DLRMs) with freshly generated data. To serve the online DLRM retraining, existing solutions use hundreds of CPU computing nodes designated for input preprocessing, causing significant power consumption that surpasses even the power usage of GPU trainers. To this end, we propose RAP, an end-to-end DLRM training framework that supports Resource-aware Automated GPU sharing for DLRM input Preprocessing and Training. The core idea of RAP is to accurately capture the remaining GPU computing resources during DLRM training for input preprocessing, achieving superior training efficiency without requiring additional resources. Specifically, RAP utilizes a co-running cost model to efficiently assess the costs of various input preprocessing operations, and it implements a resource-aware horizontal fusion technique that adaptively merges smaller kernels according to GPU availability, circumventing any interference with DLRM training. In addition, RAP leverages a heuristic searching algorithm that jointly optimizes both the input preprocessing graph mapping and the co-running schedule to maximize the end-to-end DLRM training throughput. The comprehensive evaluation shows that RAP achieves 78.3× speedup on average over CPU-based DLRM input preprocessing frameworks. In addition, the end-to-end training throughput of RAP is only 2.04% lower than the ideal case, which has no input preprocessing overhead.

Wang, Zheng↗
DOE OSTI2023

Automated, high-accuracy classification of textured microstructures using a convolutional neural network

Crystallographic texture is an important descriptor of material properties but requires time-intensive electron backscatter diffraction (EBSD) for identifying grain orientations. While some metrics such as grain size or grain aspect ratio can distinguish textured microstructures from untextured microstructures after significant grain growth, such morphological differences are not always visually observable. This paper explores the use of deep learning to classify experimentally measured textured microstructures without knowledge of crystallographic orientation. A deep convolutional neural network is used to extract high-order morphological features from binary images to distinguish textured microstructures from untextured microstructures. The convolutional neural network results are compared with a statistical Kolmogorov–Smirnov tests with traditional morphological metrics for describing microstructures. Results show that the convolutional neural network achieves a significantly improved classification accuracy, particularly at early stages of grain growth, highlighting the capability of deep learning to identify the subtle morphological patterns resulting from texture. The results demonstrate the potential of a convolutional neural network as a tool for reliable and automated microstructure classification with minimal preprocessing.

36 MATERIALS SCIENCE↗
NASA NTRS2004

Using Automated Theorem Provers to Certify Auto-Generated Aerospace Software

We describe a system for the automated certification of safety properties of NASA software. The system uses Hoare-style program verification technology to generate proof obligations which are then processed by an automated first-order theorem prover (ATP). For full automation, however, the obligations must be aggressively preprocessed and simplified We describe the unique requirements this places on the ATP and demonstrate how the individual simplification stages, which are implemented by rewriting, influence the ability of the ATP to solve the proof tasks. Experiments on more than 25,000 tasks were carried out using Vampire, Spass, and e-setheo.

Denney, Ewen↗
NASA NTRS2004

An Empirical Evaluation of Automated Theorem Provers in Software Certification

We describe a system for the automated certification of safety properties of NASA software. The system uses Hoare-style program verification technology to generate proof obligations which are then processed by an automated first-order theorem prover (ATP). We discuss the unique requirements this application places on the ATPs, focusing on automation, proof checking, and usability. For full automation, however, the obligations must be aggressively preprocessed and simplified, and we demonstrate how the individual simplification stages, which are implemented by rewriting, influence the ability of the ATPs to solve the proof tasks. Our results are based on 13 certification experiments that lead to more than 25,000 proof tasks which have each been attempted by Vampire, Spass, e-setheo, and Otter. The proofs found by Otter have been proof-checked by IVY.

Denney, Ewen↗
DOE OSTI2022

Classification of Cloud Particle Imagery from Aircraft Platforms Using Convolutional Neural Networks

Abstract A vast amount of ice crystal imagery exists from a variety of field campaign initiatives that can be utilized for cloud microphysical research. Here, nine convolutional neural networks are used to classify particles into nine regimes on over 10 million images from the Cloud Particle Imager probe, including liquid and frozen states and particles with evidence of riming. A transfer learning approach proves that the Visual Geometry Group (VGG-16) network best classifies imagery with respect to multiple performance metrics. Classification accuracies on a validation dataset reach 97% and surpass traditional automated classification. Furthermore, after initial model training and preprocessing, 10 000 images can be classified in approximately 35 s using 20 central processing unit cores and two graphics processing units, which reaches real-time classification capabilities. Statistical analysis of the classified images indicates that a large portion (57%) of the dataset is unusable, meaning the images are too blurry or represent indistinguishable small fragments. In addition, 19% of the dataset is classified as liquid drops. After removal of fragments, blurry images, and cloud drops, 38% of the remaining ice particles are largely intersecting the image border (≥10% cutoff) and therefore are considered unusable because of the inability to properly classify and dimensionalize. After this filtering, an unprecedented database of 1 560 364 images across all campaigns is available for parameter extraction and bulk statistics on specific particle types in a wide variety of storm systems, which can act to improve the current state of microphysical parameterizations.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Scalable Multi-Facility Workflows for Artificial Intelligence Applications in Climate Research

Earth observation satellites and earth system models are sources of vast, multi-modal datasets that are invaluable for advancing climate and environmental research. However, their scale and complexity pose significant challenges for processing and analysis. In this paper we discuss our experiences in developing and using a scientific research application using an automated multi-facility workflow that orchestrates data collection, preprocessing, artificial intelligence (AI) inferencing, and data movement across diverse computational resources, leveraging the Advanced Computing Ecosystem Testbed at the Oak Ridge Leadership Computing Facility (OLCF). We demonstrate that our workflow can be seamlessly integrated and orchestrated across research facilities managed by different federal agencies, thus allowing users to extract new scientific insights from climate datasets. The experimental results indicate that the multi-facility workflow significantly reduces processing time, enhances scalability, and maintains high efficiency across varying workloads. Notably, our workflow processes 12,000 high-resolution satellite images in just 44 seconds using 80 workers distributed across 10 nodes on the OLCF systems. Such high throughput is essential for dynamic tokenization and sharding of petascale satellite data for distributed AI model training and inferencing at scale across thousands of GPUs.

Kurihana, Takuya [ORNL] (ORCID:0000000156698565)↗
DOE OSTI2022

Statistical characterization of experimental magnetized liner inertial fusion stagnation images using deep-learning-based fuel–background segmentation

Significant variety is observed in spherical crystal x-ray imager (SCXI) data for the stagnated fuel–liner system created in Magnetized Liner Inertial Fusion (MagLIF) experiments conducted at the Sandia National Laboratories Z-facility. As a result, image analysis tasks involving, e.g., region-of-interest selection (i.e. segmentation), background subtraction and image registration have generally required tedious manual treatment leading to increased risk of irreproducibility, lack of uncertainty quantification and smaller-scale studies using only a fraction of available data. We present a convolutional neural network (CNN)-based pipeline to automate much of the image processing workflow. This tool enabled batch preprocessing of an ensemble of N scans = 139 SCXI images across N exp = 67 different experiments for subsequent study. The pipeline begins by segmenting images into the stagnated fuel and background using a CNN trained on synthetic images generated from a geometric model of a physical three-dimensional plasma. The resulting segmentation allows for a rules-based registration. Our approach flexibly handles rarely occurring artifacts through minimal user input and avoids the need for extensive hand labelling and augmentation of our experimental dataset that would be needed to train an end-to-end pipeline. Here we also fit background pixels using low-degree polynomials, and perform a statistical assessment of the background and noise properties over the entire image database. Our results provide a guide for choices made in statistical inference models using stagnation image data and can be applied in the generation of synthetic datasets with realistic choices of noise statistics and background models used for machine learning tasks in MagLIF data analysis. We anticipate that the method may be readily extended to automate other MagLIF stagnation imaging applications.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2022

Electron energy loss spectroscopy database synthesis and automation of core-loss edge recognition by deep-learning neural networks

Abstract The ionization edges encoded in the electron energy loss spectroscopy (EELS) spectra enable advanced material analysis including composition analyses and elemental quantifications. The development of the parallel EELS instrument and fast, sensitive detectors have greatly improved the acquisition speed of EELS spectra. However, the traditional way of core-loss edge recognition is experience based and human labor dependent, which limits the processing speed. So far, the low signal–noise ratio and the low jump ratio of the core-loss edges on the raw EELS spectra have been challenging for the automation of edge recognition. In this work, a convolutional-bidirectional long short-term memory neural network (CNN-BiLSTM) is proposed to automate the detection and elemental identification of core-loss edges from raw spectra. An EELS spectral database is synthesized by using our forward model to assist in the training and validation of the neural network. To make the synthesized spectra resemble the real spectra, we collected a large library of experimentally acquired EELS core edges. In synthesize the training library, the edges are modeled by fitting the multi-Gaussian model to the real edges from experiments, and the noise and instrumental imperfectness are simulated and added. The well-trained CNN-BiLSTM network is tested against both the simulated spectra and real spectra collected from experiments. The high accuracy of the network, 94.9%, proves that, without complicated preprocessing of the raw spectra, the proposed CNN-BiLSTM network achieves the automation of core-loss edge recognition for EELS spectra with high accuracy.

36 MATERIALS SCIENCE↗
NASA NTRS1982

Report of subpanel on feature extraction

The state of knowledge in feature extraction for Earth resource observation systems is reviewed and research tasks are proposed. Issues in the subpixel feature estimation problem are defined as: (1) the identification of image models which adequately describe the data and the sensor it is using; (2) the construction of local feature models based on those image models; and (3) the problem of trying to understand these effects of preprocessing on the entire process. The development of ground control point (GCP) libraries for automated selection presents two concerns. One is the organization of these GCP libraries for rectification problems, i.e., the problems of automatically selecting by computer the specific GCP's for particular registration tasks. Second is the importance of integrating ground control patterns in a data base management system, allowing interface to a large number of sensor image types with an automatic selection system. The development of data validation criteria for the comparison of different extraction techniques is also discussed.

Source record↗
NASA NTRS2005

Data-Acquisition Software for PSP/TSP Wind-Tunnel Cameras

Wing-Viewer is a computer program for acquisition and reduction of image data acquired by any of five different scientificgrade commercial electronic cameras used at Langley Research center to observe wind-tunnel models coated with pressure or temperature-sensitive paints (PSP/TSP). Wing-Viewer provides full automation of camera operation and acquisition of image data, and has limited data-preprocessing capability for quick viewing of the results of PSP/TSP test images. Wing- Viewer satisfies a requirement for a standard interface between all the cameras and a single personal computer: Written by use of Microsoft Visual C++ and the Microsoft Foundation Class Library as a framework, Wing-Viewer has the ability to communicate with the C/C++ software libraries that run on the controller circuit cards of all five cameras.

Amer, Tahani R.↗