Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Automatic Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

SWAP-Assembler 2: Optimization of De Novo Genome Assembler at Large Scale

In this paper, we analyze and optimize the most time-consuming steps of the SWAP-Assembler, a parallel genome assembler, so that it can scale to a large number of cores for huge genomes with the size of sequencing data ranging from terabyes to petabytes. According to the performance analysis results, the most time-consuming steps are input parallelization, k-mer graph construction, and graph simplification (edge merging). For the input parallelization, the input data is divided into virtual fragments with nearly equal size, and the start position and end position of each fragment are automatically separated at the beginning of the reads. In k-mer graph construction, in order to improve the communication efficiency, the message size is kept constant between any two processes by proportionally increasing the number of nucleotides to the number of processes in the input parallelization step for each round. The memory usage is also decreased because only a small part of the input data is processed in each round. With graph simplification, the communication protocol reduces the number of communication loops from four to two loops and decreases the idle communication time. The optimized assembler is denoted as SWAP-Assembler 2 (SWAP2). In our experiments using a 1000 Genomes project dataset of 4 terabytes (the largest dataset ever used for assembling) on the supercomputer Mira, the results show that SWAP2 scales to 131,072 cores with an efficiency of 40%. We also compared our work with both the HipMER assembler and the SWAP-Assembler. On the Yanhuang dataset of 300 gigabytes, SWAP2 shows a 3X speedup and 4X better scalability compared with the HipMer assembler and is 45 times faster than the SWAP-Assembler. The SWAP2 software is available at https://sourceforge.net/projects/swapassembler.

Meng, Jintao↗
DOE OSTI2025

VISIONARY: Virtual Intelligence System for Optimizing Novel Analytical Research Yields

VISIONARY is an AI system that accelerates energy materials discovery by automatically generating hypotheses about structure-property relationships. It analyzes patterns in materials data, identifies promising correlations, and proposes testable scientific hypotheses without human intervention. By streamlining this reasoning process, VISIONARY helps researchers efficiently identify candidate materials with desired properties, significantly speeding up the materials development pipeline for energy applications. During the project, we developed a standalone application. The application uses a combination of papers provided by the user and data collected from FutureHouse’s dataset to build an understanding of the background that the user wants to explore for the hypothesis.

36 MATERIALS SCIENCE↗
DOE OSTI2019

Estimation of Radiation Source Terms of Hanbit Unit3 Spent Fuel with Realistic Spent Fuel Data - 19043

Besides public acceptance, the construction of the dry storage facilities requires a sophisticated safety analysis for licensing, which needs accurate estimation of radiation source terms including the spent fuel inventories. In this work, we estimated the characteristics such as heat generations, radiotoxicities, and radioactivities of spent fuels discharged from HanBit Unit 3 using the recent realistic data of the spent fuels including detailed operational histories. The estimation was performed by writing an in-house program which automatically generates ORIGEN-S input files and processes their output files. Also, we performed the evaluation of source terms using these spent fuel assemblies. In particular, these results are compared with those obtained using a simple way with a single specific power and no cooling time between successive cycles. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2021

Detecting optical transients using artificial neural networks and reference images from different surveys

ABSTRACT We present a technique to detect optical transients based on an artificial neural networks method. We describe the architecture of two networks capable of comparing images of the same part of the sky taken by different telescopes. One image corresponds to the epoch in which a potential transient could exist; the other is a reference image of an earlier epoch. We use data obtained by the Dr. Cristina V. Torres Memorial Astronomical Observatory and archival reference images from the Sloan Digital Sky Survey. We trained a convolutional neural network and a dense layer network on simulated source samples and then tested the trained networks on samples created from real image data. Autonomous detection methods replace the standard process of detecting transients, which is normally achieved by source extraction of a difference image followed by human inspection of the detected candidates. Replacing the human inspection component with an entirely autonomous method would allow for a rapid and automatic follow-up of interesting targets of opportunity. The toy-model pipeline that we present here is not yet able to replace human inspection, but it might provide useful hints to identify potential candidates. The method will be further expanded and tested on telescopes participating in the Transient Optical Robotic Observatory of the South Collaboration.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Improved temperature dependence of rate coefficients for rotational state-to-state transitions in H 2 O + H 2 O collisions

Aims. We present an improved database of temperature-dependent rate coefficients for rotational state-to-state transitions in H 2 O + H 2 O collisions. The database includes 231 transitions between the lower para-states of H 2 O and 210 transitions between its lower ortho-states (up toj= 7) and can be employed in cometary and planetary applications up to the temperature of 1000 K. Methods. We developed and applied a new general method that allows the generation of rate coefficients for excitation and quenching processes that automatically satisfy the principle of microscopic reversibility and also helps to cover the range of low collision energies by interpolation of cross sections between the process threshold and the computed data points. Results. We find that in the range of intermediate temperatures, 150

Astronomy & Astrophysics↗
DOE OSTI2013

Composing Data Parallel Code for a SPARQL Graph Engine

Big data analytics process large amount of data to extract knowledge from them. Semantic databases are big data applications that adopt the Resource Description Framework (RDF) to structure metadata through a graph-based representation. The graph based representation provides several benefits, such as the possibility to perform in memory processing with large amounts of parallelism. SPARQL is a language used to perform queries on RDF-structured data through graph matching. In this paper we present a tool that automatically translates SPARQL queries to parallel graph crawling and graph matching operations. The tool also supports complex SPARQL constructs, which requires more than basic graph matching for their implementation. The tool generates parallel code annotated with OpenMP pragmas for x86 Shared-memory Multiprocessors (SMPs). With respect to commercial database systems such as Virtuoso, our approach reduces memory occupation due to join operations and provides higher performance. We show the scaling of the automatically generated graph-matching code on a 48-core SMP.

Big data↗
DOE OSTI2021

Interpretable Deep Learning for the Earth System with Fractal Nets

Focal Area 3: Explainable AI Our confidence in the projections made by Earth System Models (ESMs) depends on understanding them to be, in some important respects, faithful representations of the Earth system. Here we present an “explainable Artificial Intelligence (AI)” method that allows us to uncover the dynamical structure of the observed and modeled Earth system, discover hidden links across wide spatiotemporal scales, target model development efforts at poorly-represented dynamics, and optimize observed or modeled data collection to maximize predictive information. Science Challenge: Dynamical system science for the Earth system poses unique challenges given the large degree of internal climate variability. Thus, tools that help us understand how ESMs succeed and fail at representing these dynamics are crucial, particularly in relation to the observed system. Furthermore, the computational and memory constraints on ESM data output motivate in situ analysis of ESM dynamics, including automatic detection of dynamical shifts. Also, of key importance are procedures that leverage ESMs to optimize observational campaigns for improving process representation, reducing structural uncertainty and improving model skill.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Machine Learning-Based Anomaly Detection for PMT Data Quality Monitoring in the SBN and DUNE

Maintaining high-quality detector data is essential for achieving the scientific objectives of the Short-Baseline Neutrino (SBN) Program at Fermilab. Current data quality monitoring (DQM) procedures rely primarily on threshold-based metrics and manual inspection of detector monitoring plots, making the detection of subtle or gradually developing anomalies both time-consuming and dependent on expert interpretation. This project developed and evaluated a machine-learning workflow for automatically identifying anomalous photomultiplier tube (PMT) channels in the Short-Baseline Near Detector (SBND) using optical-hit amplitude data. A Python-based analysis program was developed to process ROOT files, extract statistical features describing individual PMT amplitude distributions, and generate feature vectors for anomaly detection. These features were used to train an Isolation Forest model using data representing normal detector operation. The trained model was subsequently applied to independent detector runs to identify channels exhibiting statistically unusual behavior relative to the learned reference response. To support expert interpretation, the workflow generated complementary diagnostic products, including anomaly score distributions, normalized amplitude comparisons, decision-tree visualizations, and principal component analysis (PCA) projections. This project demonstrated the feasibility of integrating unsupervised machine learning into detector data-quality monitoring and developed a complete workflow for automated PMT performance assessment to aid expert-driven review. Beyond its technical contributions, the VFP appointment fostered a research collaboration between Aurora University and Fermilab and provided direct workforce development benefits by training the visiting faculty member in detector-scale machine-learning methods that are now being incorporated into undergraduate coursework and research. The methodology developed here provides a foundation for future applications to ProtoDUNE and other liquid argon time projection chamber (LArTPC) detectors, contributing to ongoing efforts to improve detector reliability, reduce manual monitoring requirements, and enable scalable data quality monitoring for future large-scale neutrino experiments, including the Deep Underground Neutrino Experiment (DUNE).

Colón Santana, Juan A. [Unlisted, US, IL]↗
DOE OSTI2026

ACDC (Automated Campbell Diagram Code) [SWR-26-042]

This application provides a web-based graphical user interface to generating Campbell Diagrams and visualizing mode shapes for OpenFAST turbine models. Determining the aeroelastic stability and dynamic characteristics of wind turbines is a critical step in turbine design and analysis. Historically, extracting natural frequencies and mode shapes from OpenFAST—the industry-standard whole-turbine simulation code—has been a fragmented and tedious process. It required manual model configuration, command-line linearization execution, and complex post-processing via proprietary scripts to handle rotating-frame dynamics. To address these workflow bottlenecks, we present the Automated Campbell Diagram Code (ACDC), an open-source graphical software tool developed by the National Laboratory of the Rockies (NLR) under the DOE-funded Distributed Wind Aeroelastic Modeling (dWAM) project. ACDC streamlines the end-to-end linearization and stability analysis workflow into a single, intuitive cross-platform application. The software guides users through OpenFAST model configuration, definition of operating points, and the automated execution of steady-state trim and linearization simulations. Under the hood, ACDC automates the complex mathematical post-processing steps required for rotating systems, including Multi-Blade Coordinate (MBC) transformations, eigenanalysis, and advanced modal tracking utilizing the Modal Assurance Criterion (MAC) and spectral clustering. Finally, ACDC processes these results to automatically generate Campbell diagrams and features a robust 3D visualization engine to animate full-system mode shapes. By eliminating the reliance on external post-processing environments and manual data manipulation, ACDC significantly accelerates dynamic analysis and lowers the barrier to entry for wind energy researchers and engineers.

Summerville, Brent [National Laboratory of the Roc↗
DOE OSTI2021

AURORA: Automated Refinement of Coarse-Grained Reconfigurable Accelerators

Coarse-grained reconfigurable arrays (CGRAs), loosely defined as arrays of functional units interconnected through a network-on-chip (NoC), provide higher flexibility than domain-specific ASIC accelerators while offering increased hardware efficiency with respect to fine-grained reconfigurable devices, such as Field Programmable Gate Arrays (FPGAs). Un-fortunately, designing a CGRA for a specific application domain involves enormous software/hardware engineering effort (e.g., designing the CGRA, map operations onto the CGRA, etc) and requires the exploration on a large design space (e.g., applying appropriate loop transformation on each application, specializing the reconfigurable processing elements of the CGRA, refining the network topology, deciding the size of the data memory, etc). Int his paper, we propose AURORA – a software/hardware co-design framework to automatically synthesize optimal CGRA given a set of applications of interest

Tan, Cheng↗
DOE OSTI2018

Quantitative Mapping of Nanoscale Chemical Dynamics in Sub-sampled Operando (S)TEM Images Using Spatio-Temporal Analytics

One of the main limitations in the use of operando scanning transmission electron microscopes to study dynamic chemical processes is the effect of the electron beam on the kinetics of the reaction being observed. Here we demonstrate that a flexible Gaussian mixture model can be used to extract quantitative information directly from sub-sampled images, i.e.~images where not all the pixels in the image are illuminated with the beam. The use of this method is demonstrated for the charge/discharge cycle of a Li-battery, where the deposition and dissolution of Li at the anode can be accurately quantified for sub-sampling levels down to 1%. Performing operando imaging using a small fraction of the pixels means that the observations can significantly reduce the effect of the beam, automatically increase the imaging speed and decrease the total data transfer rate required. Such new software capabilities offer the potential to significantly widen the application of operando hardware approaches to study nanoscale dynamics in materials.

Stanfill, Bryan A.↗
DOE OSTI2017

Multi-static Serial LiDAR for Surveillance and Identification of Marine Life at MHK Installations

Final Report for project DE-EE0006787: Multi-static Serial LiDAR for Surveillance and Identification of Marine Life at MHK Installations. This project developed and tested an optical monitoring system prototype that will be suitable for marine and hydrokinetic (MHK) full project lifecycle observation (baseline, commissioning, and decommissioning), with automated real-time classification of marine animals. This system can be deployed to collect pre-installation baseline species observations at a proposed deployment site with minimal post-processing overhead. To satisfy deployed MHK project species of concern (e.g. Endangered Species Act-listed) monitoring requirements, the system provides automated tracking and notification of the presence of managed animals within established perimeters of MHK equipment and provides high resolution imagery of their behavior through a wide range of conditions. During a project’s decommissioning stage, the system can remain installed to provide resource managers with post-installation data. Our technology, known as an Unobtrusive Multi-static Serial LiDAR Imager (UMSLI), is a technology transfer of underwater distributed LiDAR imaging technology that preserves the advantages of traditional optical and acoustic solutions while overcoming associated disadvantages for MHK environmental monitoring applications. This new approach is a purposefully-designed, reconfigurable adaptation of an existing technology that can be easily mounted on or around different classes of MHK equipment. The system uses low average power red (638nm) laser illumination to be invisible and eye-safe to marine animals and is compact and cost effective. The equipment is designed for long term, maintenance-free operations, to inherently generate a sparse primary dataset that only includes detected anomalies (animal presence information), and to allow robust real-time automated animal classification/identification with a low data bandwidth requirement. Advantages of the technology over others currently being used or being considered for MHK monitoring include: Unlike a conventional camera, the depth of field is near-infinite and limited by attenuation (approximately 5-8 m) rather than focal properties of a lens; Operation in an adaptive mode which can project a sparse grid of pulses with higher peak power for longer range detection (>10 meters) and track animals within a zone of interest with high resolution imagery for identification of marine life at closer range (<5m); System detection limit and Signal-to-Noise-Ratio is superior to a camera, due to rejection of both backscattering component and ambient solar background; Multiple wide-angle pulsed laser illuminators and bucket detectors can be flexibly configured to cover a 4pi steradian (i.e. omnidirectional) scene volume, while also retrieving 3D features of animal targets from timing information; Process and classification framework centered around a novel active learning and incremental classification classifier that enables accurate identification of a variety of marine animals automatically; A two-tiered monitoring architecture and invisible watermarking-based data archiving and retrieving approach ensures significant data reduction while preserving high fidelity monitoring. A methodology to train and optimize the classifier for target species of concern to optimize site monitoring effectiveness. This technological innovation addresses a high priority regulatory requirement to observe marine life interaction near MHK projects. Our solution improves resource manager confidence that any interactions between marine animals and equipment are observed in a cost-effective and automated manner. Without EERE funding, this novel application of multi-static LiDAR would not have been available to the MHK community for environmental monitoring.

16 TIDAL AND WAVE POWER↗
DOE OSTI2022

3DBFSVBF (3D BatFinder Smart Video BioFilter and Multi-class BatFinder Smart Video BioFilter) [SWR-22-88]

Bats are notoriously difficult to study, therefore, identifying specific behavioral trends and the precise environmental conditions at the time of collision requires a monitoring solution that can reliably collect relevant data. To date, thermal infrared video surveillance has been extensively applied to study bats and has proven to be a powerful yet cumbersome tool. Current analytical approaches are time consuming because data processing data has not been fully automated. In the past, steps have been taken to record avian and bat activity in conjunction with complicated image processing techniques that separate species from other moving objects within the field of view (i.e. clouds and portions of the wind turbine). Once the videos are collected, the post-processing does not allow real time monitoring and identification, leading to a delay in both studying the behavior of these species and determining the effectiveness of any impact reduction strategy being studied. Moreover, object identification capability is lacking, thus limiting the usefulness of video data. To resolve these issues, we are using open source 3D computer vision and machine learning techniques allowing for automatic detection of objects in real-time with the ability to correlate these objects with environmental variables and recording the flight paths of each object. The machine learning has been trained on 3D data and allows for automated real-time data collection, identification and tracking, thereby eliminating the need for long and tedious post-analysis processing of the videos. This machine learning model is an added feature to the previous BatFinder Smart Video BioFilter and increases the accuracy of that systems classification by increasing the accuracy of identifying bats (90% accuracy) and insects (69% accuracy) to a 97% accuracy. There are two object classifier machine learning models, Binary and multi-classification. Binary object classifier labeled BatFinder_Smart_Video_BioFilter.h5 distinguishes between biological objects and non-biological objects. The main goal of this object classifier is to ignore the turbine blades while detecting biological object flying withing the rotor swept area of the turbine. Non-biological objects have a probability of 0 and biological objects have a probability of 1. Multi-classifier labeled Multiclass_BatFinder_Smart_Video_BioFilter.h5 distinguishes between bats, birds, insects and non-biological.

Yarbrough, John↗
DOE OSTI2018

A new MR-SAD algorithm for the automatic building of protein models from low-resolution X-ray data and a poor starting model

Determining macromolecular structures from X-ray data with resolution worse than 3Å remains a challenge. Even if a related starting model is available, its incompleteness or its bias together with a low observation-to-parameter ratio can render the process unsuccessful or very time-consuming. Yet, many biologically important macromolecules, especially large macromolecular assemblies, membrane proteins and receptors, tend to provide crystals that diffract to low resolution. A new algorithm to tackle this problem is presented that uses a multivariate function to simultaneously exploit information from both an initial partial model and low-resolution single-wavelength anomalous diffraction data. The new approach has been used for six challenging structure determinations, including the crystal structures of membrane proteins and macromolecular complexes that have evaded experts using other methods, and large structures from a 3.0 Å resolution F1-ATPase data set an a 4.5Å resolution SecYEG–SecA complex data set. All of the models were automatically built by the method to Rfree values of between 28.9 and 39.9% and were free from the initial model bias.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

SolarAPP+ Performance Review (2023 Data)

The Solar Automated Permit Processing Plus (SolarAPP+) platform is an online portal to facilitate and expedite rooftop solar photovoltaic (PV) and battery storage permitting processes. SolarAPP+ allows PV contractors to upload system specifications, have that information automatically reviewed for code compliance, and receive instant approval for code-compliant systems, reducing authority having jurisdiction (AHJ) staff time needed for review. SolarAPP+ also provides inspection checklists to verify installation practices and adherence to approved designs. SolarAPP+ is available to AHJs at no cost. This report is part of an ongoing series of reviews of SolarAPP+ performance. Consistent with previous performance reviews, we summarize SolarAPP+ adoption trends to date and compare various metrics for PV systems permitted through SolarAPP+ versus systems permitted through traditional AHJ permitting processes. As of the end of 2023, the National Renewable Energy Laboratory (NREL) had contacted over 1,700 AHJs with significant solar permitting volume regarding SolarAPP+. Of those, 793 AHJs had expressed interest in the platform as of the end of 2023. 161 AHJs had begun piloting the platform and 97 of these had publicly launched the platform by the end of 2023. In 2023, 668 installers submitted 18,906 permits through the SolarAPP+ platform, including 4,834 permits submitted as part of a solar plus storage program. SolarAPP+ permits accounted for around 43% of all permits issued in participating AHJs. We compare permitting timelines through SolarAPP+ to traditional AHJ permitting processes to assess the platform's performance. Consistent with previous SolarAPP+ performance reviews, we find that permitting timelines are significantly shorter for SolarAPP+ projects. Based on median timelines, a typical SolarAPP+ project is permitted and inspected 14.5 business days sooner than traditional projects. We estimate that automatic SolarAPP+ permitting saved around 7,200 hours of AHJ staff time in 2023. Finally, we estimate that SolarAPP+ eliminated over 150,000 business days in permitting-related delays in 2023.

14 SOLAR ENERGY↗
DOE OSTI2026

SolarAPP+ Performance Review (2024 Data)

The Solar Automated Permit Processing Plus (SolarAPP+) platform is an online portal to facilitate and expedite rooftop solar photovoltaic (PV) and battery storage permitting processes. SolarAPP+ allows PV contractors to upload system specifications, have that information automatically reviewed for code compliance, and receive instant approval for code-compliant systems, reducing authority having jurisdiction (AHJ) staff time needed for review. SolarAPP+ also provides inspection checklists to verify installation practices and adherence to approved designs. This report is part of an ongoing series of reviews of SolarAPP+ performance. Consistent with previous performance reviews, we summarize SolarAPP+ adoption trends to date and compare various metrics for PV systems permitted through SolarAPP+ versus systems permitted through traditional AHJ permitting processes. As of the end of 2024, 799 AHJs had expressed interest in the platform, with 264 fully adopting (215) or piloting (49) the platform. In 2024, 861 installers submitted 37,393 permits through the SolarAPP+ platform, including 27,375 permits for PV+storage systems. SolarAPP+ permits accounted for around 43% of all permits issued in all participating AHJs, and more than 60% of all permits in several participating AHJs. We compare permitting timelines through SolarAPP+ to traditional AHJ permitting processes to assess the platform's performance. Consistent with previous SolarAPP+ performance reviews, we find that permitting timelines are significantly shorter for SolarAPP+ projects. Based on median timelines, a typical SolarAPP+ project is permitted and inspected 12 business days sooner than traditional projects. We estimate that automatic SolarAPP+ permitting saved around 18,400 hours of AHJ staff time in 2024. Finally, we estimate that SolarAPP+ eliminated over 100,000 business days in permitting-related delays in 2024.

14 SOLAR ENERGY↗
DOE OSTI2021

Rapid Automated Annotation and Analysis of N-Glycan Mass Spectrometry Imaging Data Sets Using NGlycDB in METASPACE

Imaging N-glycan spatial distribution in tissues using mass spectrometry imaging (MSI) is emerging as a promising tool in biological and clinical applications. However, there is currently no high throughput tool for visualization and molecular annotation of N-glycans in MSI data, which significantly slows down data processing and hampers the applicability of this approach. In this work, we present how METASPACE, an open-source cloud engine for molecular annotation of MSI data, can be used to automatically annotate, visualize, analyze, and interpret high-resolution mass spectrometry-based spatial N-glycomics data. METASPACE is an emerging tool in spatial metabolomics, but the lack of compatible glycan databases has limited its application for comprehensive N-glycan annotations from MSI datasets. We created NGlycDB, a public database of N-glycans, by adapting available glycan databases. We demonstrate the applicability of NGlycDB in METASPACE by analyzing MALDI-MSI data from formalin-fixed paraffin-embedded (FFPE) human kidney and mouse lung tissue sections. We added NGlycDB to METASPACE for public use, thus facilitating applications of MSI in glycobiology.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2016

On the Impact of Widening Vector Registers on Sequence Alignment

Vector extensions, such as SSE, have been part of the x86 since the 1990s, with applications in graphics, signal processing, and scientific applications. Although many algorithms and applications can naturally benefit from automatic vectorization techniques, there are still many that are difficult to vectorize due to their dependence on irregular data structures, dense branch operations, or data dependencies. Sequence alignment, one of the most widely used operations in bioinformatics workflows, has a computational footprint that features complex data dependencies. In this paper, we demonstrate that the trend of widening vector registers adversely affects the state-of-the-art sequence alignment algorithm based on striped data layouts. We present a practically efficient SIMD implementation of a parallel scan based sequence alignment algorithm that can better exploit wider SIMD units. We conduct comprehensive workload and use case analyses to characterize the relative behavior of the striped and scan approaches and identify the best choice of algorithm based on input length and SIMD width.

Needleman-Wunsch↗