Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Preprocessing Systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

DICER: Data Intensive Computing Environment and Runtime for Evaluating Unprecedented Scale of Geospatial-Temporal Human Mobility Data

With the significant increase in sources and volume of human mobility data through commercial data vendors as well as microsimulation of cities, the scale of geospatial-temporal data to analyze and assess for mobility characterization has grown to the level of Big Data. There are mobility related commercial organizations deploying scalable computing, but often the system architecture, workflow, and intermediate processing components are not fully disclosed in relevant scope. Current research literature has a notable lack of studies demonstrating architectures and workflows for human mobility analytics that are implemented on a TeraByte scale of geospatial-temporal data. In this context, this paper presents a hyperscale-level system solution named DICER (Data Intensive Computing Environment and Runtime) for processing and analytics of geospatial-temporal data at big data scale. Although the cluster computing architecture of DICER with Apache Spark job running on Kubernetes cluster is not new, there are innovations in the workflow, hierarchical processing logic, and a wide range of intermediate preprocessing and mobility metrics calculation. We have performed case studies to validate the effectiveness of DICER system solution by performing detailed analytics and assessment of human mobility microsimulation output at three different scopes and scale, including a usecase with 16.97 TeraByte and 259.2 Billion rows of data. In addition, we have presented another case study of utilizing DICER to perform the same mobility processing and comparative analytics on large-scale commercially available geospatial-temporal data. All these case studies validate the efficiency and usefulness of DICER in computing population mobility characteristics from geospatial-temporal trajectory data at an unprecedented scale (not only just data volume, but also combination of: number of user entities, temporal frequency, spatial resolution, data duration).

De, Debraj↗
DOE OSTI2023

Chapter Four—MSW characterization and preprocessing for biofuels and bioproducts

The organic fraction of MSW (OFMSW) has a high carbohydrate and lipid content, indicating its good potential as a feedstock for bioethanol and biodiesel production. However, variability in feedstock composition for MSW is a critical challenge for downstream conversion processes such as pyrolysis, gasification, incineration, and biotechnological conversion. MSW characterization is critical to inform and optimize the design of physical handling processes to minimize costs and environmental impacts, and improve the downstream conversion performance of MSW materials. Here, this book chapter focuses on current MSW characterization and preprocessing technologies and discusses the costs, benefits, and environmental impacts associated with MSW preprocessing. Also, the book chapter outlines the potential strategies to increase total revenue and mitigate negative environmental impacts for MSW recycling systems.

09 BIOMASS FUELS↗
DOE OSTI2024

Applications of LIF to Document Natural Variability of Chlorophyll Content and Cu Uptake in Moss

Chlorophyll has long been used as a natural indicator of plant health and photosynthetic efficiency. Laser-induced fluorescence (LIF) is an emerging technique for understanding broad spectrum organic processes and has more recently been used to monitor chlorophyll response in plants. Previous work has focused on developing a LIF technique for imaging moss mats to identify metal contamination with the current focus shifting toward application to moss fronds and aiding sample collection for chemical analysis. Two laser systems (CoCoBi a Nd:YGa pulsed laser system and Chl-SL with two blue continuous semiconductor diodes) were used to collect images of moss fronds exposed to increasing levels of Cu (1, 10, and 100 nmol/cm 2 ) using a CMOS camera. The best methods for the preprocessing of images were conducted before the analysis of fluorescence signatures were compared to a control. The Chl-SL system performed better than the CoCoBi, with dynamic time warping (DTW) proving the most effective for image analysis. Manual thresholding to remove lower decimal code values improved the data distributions and proved whether using one or two fronds in an image was more advantageous. A higher DTW difference from the control correlated to lower chlorophyll a/b ratios and a higher metal content, indicating that LIF, with the aid of image processing, can be an effective technique for identifying Cu contamination shortly after an event.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Pilot Plant Reliability Metrics for Grinding and Fast Pyrolysis of Woody Residues

The reliability of biomass size reduction, feeding, and conversion processes remains a critical technical hurdle and important cost driver for biomass conversion systems. While Biorefinery projects have typically focused on scaling up conversion technologies that maximize yield of desired products, e.g. fuels and chemicals, solids handling and reactor feeding systems are often considered secondarily. These are frequently scaled down or adapted from other industries, without a full appreciation of how biomass heterogeneity and variability impact process reliability. Here we present the effects of feedstock variability on relative throughput, mass loss, plant uptime, operator intervention, product yield, and product quality for grinding and fast pyrolysis operations at the pilot scale. Loblolly pine residues were sourced and collected using commercial harvesting techniques. Grinding and pyrolysis experiments were performed at INL's Biomass Feedstock National User Facility and NREL's Thermal and Catalytic Process Development Unit, respectively. Preprocessing throughput using a hammer mill varied between 31-48% of nameplate capacity (5 tons/hr), and fast pyrolysis throughput varied between 57-72% of nameplate capacity (20 kg/hr) for an overall system throughput of 18-35% of nameplate capacity across biomass preprocessing and conversion to pyrolysis oil. Pyrolysis oil yields varied between 46-53% (feedstock carbon to oil, dry basis). Grinder overloads in the size reduction step were more prevalent for lower ash and higher moisture materials. During fast pyrolysis operations, downtime was caused by bridging in the feed and char removal systems and plugging in the condensation system, and significant variation in the type and frequency of operator intervention was observed between feedstocks. Cohesion of solids (feedstock and char) leading to system plugging appeared less problematic for higher ash feedstocks, and differences in condenser plugging behavior between high and low ash feedstocks indicates the composition and physical nature of the deposited material is important for process reliability and is feedstock dependent.

09 BIOMASS FUELS↗
DOE OSTI2024

Survey of Use Cases and Scenarios on the Open Energy Data Initiative Solar Systems Integration (OEDI SI) Platform

The Open Energy Data Initiative Solar Systems Integration (OEDI SI) Data and Modeling Platform offers a comprehensive set of use cases tailored for power systems analysis. Each use case is centered around a specific power system analysis problem, supported by composite input data and reference algorithms. These composite input datasets are meticulously assembled using OEDI SI's data preprocessing tools, which integrate raw data from various sources. The primary objectives of the OEDI SI Platform include facilitating access to composite input data through widely accepted input/output formats and verified results. This accessibility enables power system network researchers and developers to validate their algorithms and showcase their applications' capabilities to the broader community. Moreover, the platform strives to promote reproducible, robust, replicable, and generalizable solar systems integration research.

14 SOLAR ENERGY↗
DOE OSTI2021

Herbaceous Feedstock 2021 State of Technology Report

The U.S. Department of Energy (DOE) promotes the production of advanced liquid transportation fuels from lignocellulosic biomass by funding fundamental and applied research that advances the State of Technology (SOT). As part of its involvement with this mission, Idaho National Laboratory (INL) completes an annual SOT report for biomass feedstock logistics. This report summarizes supply system impacts of Bioenergy Technologies Office (BETO)-funded research and development efforts at INL and INL collaboration with external partners (e.g. Forest Concepts, Purdue University ) that lead to improvements in feedstock supply systems. These include improvements to and observed performance of innovative harvest and collection methods, storage technologies, transportation and handling approaches, and advanced preprocessing technologies. Biomass quality and variability, and the interface between feedstock quality and conversion performance are key drivers in addition to delivered feedstock cost. In this report, we estimate the benefits of R&D technology improvements to individual supply system unit operations and present the status of feedstock logistics technology development for converting herbaceous biomass into biofuels. These analyses are supported by experimental data where possible and help to align the SOT relative to the cost goals defined in the Multi-Year Plan.

09 BIOMASS FUELS↗
DOE OSTI2024

Updates to the Hanford Soil Inventory Model (SIM Version 2) for FY 2023

The purpose of this environmental calculation file (ECF) is to document the updates made to the Soil Inventory Model version 2.1 (SIM-v2.1) (ECF-HANFORD-21-0073, Updates to the Hanford Soil Inventory Model (SIM Version 2) for FY 2021) in FY 2023. These updates to SIM-v2.1 are referred to as the SIM-v2.2 version. The following updates have been made: • Include inventory estimate for a new analyte • hexavalent chromium (Cr(VI)), separate from total chromium inventory • Revise the inventory discharged to the 216-U-10 and 216-T-4 Pond systems based on partitioning of waste streams discharges over time and space among influent ditches and ponds • Enhance the preprocessing and postprocessing of the input and output files Note that the methodology for estimating Cr(VI) inventories is based on ECF-200W-23-0040, Recommendations for Updating Liquid Discharged Inventory and Transport Modeling Parameters for Cumulative Impacts Evaluation of Hexavalent Chromium in the 200 West Area while the revision of inventory discharged to the 216-U-10 and 216-T-4 Pond systems is based on ECF-HANFORD-19-0032, Distribution of Infiltration in the 216-U-10, 216-B-3 Pond, and 216-T-4 Pond Systems 1944-1997.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Technical Language Processing of Nuclear Power Plants Equipment Reliability Data

Operating nuclear power plants (NPPs) generate and collect large amounts of equipment reliability (ER) element data that contain information about the status of components, assets, and systems. Some of this information is in textual form where the occurrence of abnormal events or maintenance activities are described. Analyses of NPP textual data via natural language processing (NLP) methods have expanded in the last decade, and only recently the true potential of such analyses has emerged. So far, applications of NLP methods have been mostly limited to classification and prediction in order to identify the nature of the given textual element (e.g., safety or non-safety relevant). In this paper, we target a more complex problem: the automatic generation of knowledge based on a textual element in order to assist system engineers in assessing an asset’s historical health performance. The goal is to assist system engineers in the identification of anomalous behaviors, cause–effect relations between events, and their potential consequences, and to support decision-making such as the planning and scheduling of maintenance activities. “Knowledge extraction” is a very broad concept whose definition may vary depending on the application context. In our particular context, it refers to the process of examining an ER textual element to identify the systems or assets it mentions and the type of event it describes (e.g., component failure or maintenance activity). In addition, we wish to identify details such as measured quantities and temporal or cause–effect relations between events. This paper describes how ER textual data elements are first preprocessed to handle typos, acronyms, and abbreviations, then machine learning (ML) and rule-based algorithms are employed to identify physical entities (e.g., systems, assets, and components) and specific phenomena (e.g., failure or degradation). A few applications relevant from an NPP ER point of view are presented as well.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

TomoPyUI : a user-friendly tool for rapid tomography alignment and reconstruction

The management and processing of synchrotron and neutron computed tomography data can be a complex, labor-intensive and unstructured process. Users devote substantial time to both manually processing their data ( i.e. organizing data/metadata, applying image filters etc. ) and waiting for the computation of iterative alignment and reconstruction algorithms to finish. In this work, we present a solution to these problems: TomoPyUI , a user interface for the well known tomography data processing package TomoPy . This highly visual Python software package guides the user through the tomography processing pipeline from data import, preprocessing, alignment and finally to 3D volume reconstruction. The TomoPyUI systematic intermediate data and metadata storage system improves organization, and the inspection and manipulation tools (built within the application) help to avoid interrupted workflows. Notably, TomoPyUI operates entirely within a Jupyter environment. Herein, we provide a summary of these key features of TomoPyUI , along with an overview of the tomography processing pipeline, a discussion of the landscape of existing tomography processing software and the purpose of TomoPyUI , and a demonstration of its capabilities for real tomography data collected at SSRL beamline 6-2c.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Common Column Identification for Table Similarity Detection in Electrified Transportation Data Lakes

Electrified transportation often requires researchers and operators to interact with datasets from a wide range of sources and disciplines, such as transportation, power systems, public health, policies, and regulations. These datasets vary in quality and format, making it difficult to understand, preprocess, and identify key columns representing real-world entities or values for indexing and joining, which can negatively impact downstream analysis and operation. Existing solutions are limited, requiring extensive manual customization or data expertise to utilize. In this article, we propose a multi-layered approach to automatically identify key columns to expedite preprocessing and aid in analysis of electrified transportation data. Our method leverages a dynamic ontology to identify common fields and an information theory-based strategy for edge cases that are difficult to generalize. Evaluations on a number of datasets from data.gov and kaggle.com show improved performance of our methods over several baseline techniques, and our ablation analyses illustrate the efficacy of individual components of our method. Our case studies also demonstrate that our methods have the potential to improve analysis of electrified transportation data and aid in automatic integration of such datasets.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2020

Autonomous Tuning and Charge-State Detection of Gate-Defined Quantum Dots

Defining quantum dots in semiconductor-based heterostructures is an essential step in initializing solid-state qubits. With growing device complexity and increasing number of functional devices required for measurements, a manual approach to finding suitable gate voltages to confine electrons electrostatically is impractical. Here, we implement a two-stage device characterization and dot-tuning process, which first determines whether devices are functional and then attempts to tune the functional devices to the single or double quantum-dot regime. We show that automating well-established manual-tuning procedures and replacing the experimenter’s decisions by supervised machine learning is sufficient to tune double quantum dots in multiple devices without premeasured input or manual intervention. The quality of measurement results and charge states are assessed by four binary classifiers trained with experimental data, reflecting real device behavior. We compare and optimize eight models and different data preprocessing techniques for each of the classifiers to achieve reliable autonomous tuning, an essential step towards scalable quantum systems in quantum-dot-based qubit architectures.

Darulova, J↗
DOE OSTI2021

Feedstock Harvesting and Storage: Post-Harvest Management for Quality Preservation

Research on harvesting, collection, and storage has matured, and our understanding of feedstock supply challenges has led us to the point of our current goals. The goals of this project include using storage time to collect impactful data of critical feedstock properties in storage and to characterize the feedstock before delivery. This data will inform delivery scheduling of biomass and relevant compositional information to minimize variations in the material quality, which are critical to the preprocessing and conversion performance.

09 BIOMASS FUELS↗
DOE OSTI2021

Preprocessing for Unintended Conducted Emissions Classification with ResNet

Characterization of Unintended Conducted Emissions (UCE) from electronic devices is important when diagnosing electromagnetic interference, performing nonintrusive load monitoring (NILM) of power systems, and monitoring electronic device health, among other applications. Prior work has demonstrated that UCE analysis can serve as a diagnostic tool for energy efficiency investigations and detailed load analysis. While explaining the feature selection of deep networks with certainty is often not fully comprehensive, or in other applications, quite lacking, additional tools/methods for further corroboration and confirmation can help further the understanding of the researcher. This is true especially in the subject application of the study in this paper. Often the focus of such efforts is the selected features themselves, and there is not as much understanding gained about the noise in the collected data. If selected feature and noise characteristics are known, it can be used to further shape the design of the deep network or associated preprocessing. This is additionally difficult when the available data are limited, as in the case which the authors investigated in this study. Here, the authors present a novel work (which is a proposed complementary portion of the overall solution to the deep network classification explainability problem for this application) by applying a systematic progression of preprocessing and a deep neural network (ResNet architecture) to classify UCE data obtained via current transformers. By using a methodical application of preprocessing techniques prior to a deep classifier, hypotheses can be produced concerning what features the deep network deems important relative to what it perceives as noise. For instance, it is hypothesized in this particular study as a result of execution of the proposed method and periodic inspection of the classifier output that the UCE spectral features are relatively close to each other or to the interferers, as systematically reducing the beta parameter of the Kaiser window produced progressively better classification performance, but only to a point, as going below the Beta of eight produced decreased classifier performance, as well as the hypothesis that further spectral feature resolution was not as important to the classifier as rejection of the leakage from a spectrally distant interference. This can be very important in unpredictable low-FNR applications, where knowing the difference between features and noise is difficult. As a side-benefit, much was learned regarding the best preprocessing to use with the selected deep network for the UCE collected from these low power consumer devices obtained via current transformers. Baseline rectangular windowed FFT preprocessing provided a 62% classification increase versus using raw samples. After performing a more optimal preprocessing, more than 90% classification accuracy was achieved across 18 low-power consumer devices for scenarios in which the in-band features-to-noise ratio (FNR) was very poor.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

A portable application framework for energy management and information systems (EMIS) solutions using Brick semantic schema

This paper introduces a portable framework for developing, scaling and maintaining energy management and information systems (EMIS) applications using an ontology-based approach. Key contributions include an interoperable layer based on Brick schema, the formalization of application constraints pertaining metadata and data requirements, and a field demonstration. The framework allows for querying metadata models, fetching data, preprocessing, and analyzing data, thereby offering a modular and flexible workflow for application development. Its effectiveness is demonstrated through a case study involving the development and implementation of a data-driven anomaly detection tool for the photovoltaic systems installed at the Politecnico di Torino, Italy. During eight months of testing, the framework was used to tackle practical challenges including: (i) developing a machine learning-based anomaly detection pipeline, (ii) replacing data-driven models during operation, (iii) optimizing model deployment and retraining, (iv) handling critical changes in variable naming conventions and sensor availability (v) extending the pipeline from one system to additional ones.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2020

Improving climate model coupling through a complete mesh representation: a case study with E3SM (v1) and MOAB (v5.x)

One of the fundamental factors contributing to the spatiotemporal inaccuracy in climate modeling is the mapping of solution field data between different discretizations and numerical grids used in the coupled component models. The typical climate computational workflow involves evaluation and serialization of the remapping weights during the preprocessing step, which is then consumed by the coupled driver infrastructure during simulation to compute field projections. Tools like Earth System Modeling Framework (ESMF) and TempestRemap offer capability to generate conservative remapping weights, while the Model Coupling Toolkit (MCT) that is utilized in many production climate models exposes functionality to make use of the operators to solve the coupled problem. However, such multistep processes present several hurdles in terms of the scientific workflow and impede research productivity. In order to overcome these limitations, we present a fully integrated infrastructure based on the Mesh Oriented datABase (MOAB) library, which allows for a complete description of the numerical grids and solution data used in each submodel. Through a scalable advancing-front intersection algorithm, the supermesh of the source and target grids are computed, which is then used to assemble the high-order, conservative, and monotonicity-preserving remapping weights between discretization specifications. The Fortran-compatible interfaces in MOAB are utilized to directly link the submodels in the Energy Exascale Earth System Model (E3SM) to enable online remapping strategies in order to simplify the coupled workflow process. We demonstrate the superior computational efficiency of the remapping algorithms in comparison with other state-of-the-science tools and present strong scaling results on large-scale machines for computing remapping weights between the spectral element atmosphere and finite volume discretizations on the polygonal ocean grids.

58 GEOSCIENCES↗
DOE OSTI2026

The Impact of Time-Aware Design Choices in ICS Anomaly Detection

Industrial control systems (ICS) remain vulnerable to increasingly sophisticated cyberattacks, yet evaluating anomaly detection models in these environments is challenging due to temporal dependencies, missing-not-at-random patterns, and extremely imbalanced datasets. These factors make common practices—especially random data splits and na¨ıve imputation— prone to severe temporal leakage, which can inflate reported performance and obscure real-world limitations. In this work, we systematically examine classical machine learning models, temporal deep learning architecture, and tensordecomposition– based methods on a gas-pipeline dataset using a fully temporally separated evaluation pipeline designed to mimic realistic deployment conditions. Our findings show that proper temporal handling and MNAR-aware preprocessing significantly alter the relative performance of popular anomaly-detection methods, providing practical guidance for designing reliable, leakage-resistant ICS intrusion-detection systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Design Choices in Anomaly Detection for Industrial Control Systems: Insights from Gas Pipeline Data

Industrial control systems (ICS) remain vulnerable to increasingly sophisticated cyberattacks, yet evaluating anomaly detection models in these environments is challenging due to temporal dependencies, missing-not-at-random patterns, and extremely imbalanced datasets. These factors make common practices—especially random data splits and naïve imputation—prone to severe temporal leakage, which can inflate reported performance and obscure real-world limitations. In this work, we systematically examine classical machine learning models, temporal deep learning architecture, and tensor-decomposition–based methods on a gas-pipeline dataset using a fully temporally separated evaluation pipeline designed to mimic realistic deployment conditions. Our findings show that proper temporal handling and MNAR-aware preprocessing significantly alter the relative performance of popular anomaly-detection methods, providing practical guidance for designing reliable, leakage-resistant ICS intrusion-detection systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

AD – Elog Data Search Using Natural Language Processing Techniques

The goal of this project is to develop and evaluate a Machine Learning model using natural language processing techniques to categorize AD E-Log entries efficiently. AD E-Logs, which play a vital role in Fermilab's projects as a system to record and manage various data, often suffer from slow search processing and imprecise results. To address this issue, we preprocessed the data and employed the Doc2Vec model for training. The resulting model enables us to identify and retrieve the most relevant entries, thus enhancing user experience in accessing desired information.

Muse, Amiin↗