SEARCH · Engineering Papers

Results for “Automatic Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

All sources NASA NTRS DOE OSTI

DOE OSTI2016

On the Impact of Widening Vector Registers on Sequence Alignment

Vector extensions, such as SSE, have been part of the x86 since the 1990s, with applications in graphics, signal processing, and scientific applications. Although many algorithms and applications can naturally benefit from automatic vectorization techniques, there are still many that are difficult to vectorize due to their dependence on irregular data structures, dense branch operations, or data dependencies. Sequence alignment, one of the most widely used operations in bioinformatics workflows, has a computational footprint that features complex data dependencies. In this paper, we demonstrate that the trend of widening vector registers adversely affects the state-of-the-art sequence alignment algorithm based on striped data layouts. We present a practically efficient SIMD implementation of a parallel scan based sequence alignment algorithm that can better exploit wider SIMD units. We conduct comprehensive workload and use case analyses to characterize the relative behavior of the striped and scan approaches and identify the best choice of algorithm based on input length and SIMD width.

Needleman-Wunsch↗

DOE OSTI2015

Pairwise Sequence Alignment Library

Vector extensions, such as SSE, have been part of the x86 CPU since the 1990s, with applications in graphics, signal processing, and scientific applications. Although many algorithms and applications can naturally benefit from automatic vectorization techniques, there are still many that are difficult to vectorize due to their dependence on irregular data structures, dense branch operations, or data dependencies. Sequence alignment, one of the most widely used operations in bioinformatics workflows, has a computational footprint that features complex data dependencies. The trend of widening vector registers adversely affects the state-of-the-art sequence alignment algorithm based on striped data layouts. Therefore, a novel SIMD implementation of a parallel scan-based sequence alignment algorithm that can better exploit wider SIMD units was implemented as part of the Parallel Sequence Alignment Library (parasail). Parasail features: Reference implementations of all known vectorized sequence alignment approaches. Implementations of Smith Waterman (SW), semi-global (SG), and Needleman Wunsch (NW) sequence alignment algorithms. Implementations across all modern CPU instruction sets including AVX2 and KNC. Language interfaces for C/C++ and Python.

Jeff Daily, PNNL↗

DOE OSTI2015

Pairwise Sequence Alignment Library

Jeff Daily, PNNL↗

DOE OSTI2017

Divide and Conquer (DC) BLAST: fast and easy BLAST execution within HPC environments

Bioinformatics is currently faced with very large-scale data sets that lead to computational jobs, especially sequence similarity searches, that can take absurdly long times to run. For example, the National Center for Biotechnology Information (NCBI) Basic Local Alignment Search Tool (BLAST and BLAST+) suite, which is by far the most widely used tool for rapid similarity searching among nucleic acid or amino acid sequences, is highly central processing unit (CPU) intensive. While the BLAST suite of programs perform searches very rapidly, they have the potential to be accelerated. In recent years, distributed computing environments have become more widely accessible and used due to the increasing availability of high-performance computing (HPC) systems. Therefore, simple solutions for data parallelization are needed to expedite BLAST and other sequence analysis tools. However, existing software for parallel sequence similarity searches often requires extensive computational experience and skill on the part of the user. In order to accelerate BLAST and other sequence analysis tools, Divide and Conquer BLAST (DCBLAST) was developed to perform NCBI BLAST searches within a cluster, grid, or HPC environment by using a query sequence distribution approach. Scaling from one (1) to 256 CPU cores resulted in significant improvements in processing speed. Thus, DCBLAST dramatically accelerates the execution of BLAST searches using a simple, accessible, robust, and parallel approach. DCBLAST works across multiple nodes automatically and it overcomes the speed limitation of single-node BLAST programs. DCBLAST can be used on any HPC system, can take advantage of hundreds of nodes, and has no output limitations. Thus, this freely available tool simplifies distributed computation pipelines to facilitate the rapid discovery of sequence similarities between very large data sets.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2016

SU-G-BRB-04: Automated Output Factor Measurements Using Continuous Data Logging for Linac Commissioning

Purpose: Linac commissioning is a time consuming and labor intensive process, the streamline of which is highly desirable. In particular, manual measurement of output factors for a variety of field sizes and energy greatly hinders the commissioning efficiency. In this study, automated measurement of output factors was demonstrated as ‘one-click’ using data logging of an electrometer. Methods: Beams to be measured were created in the recording and verifying (R&V) system and configured for continuous delivery. An electrometer with an automatic data logging feature enabled continuous data collection for all fields without human intervention. The electrometer saved data into a spreadsheet every 0.5 seconds. A Matlab program was developed to analyze the excel data to monitor and check the data quality. Results: For each photon energy, output factors were measured for five configurations, including open field and four wedges. Each configuration includes 72 fields sizes, ranging from 4×4 to 20×30 cm{sup 2}. Using automation, it took 50 minutes to complete the measurement of 72 field sizes, in contrast to 80 minutes when using the manual approach. The automation avoided the necessity of redundant Linac status checks between fields as in the manual approach. In fact, the only limiting factor in such automation is Linac overheating. The data collection beams in the R&V system are reusable, and the simplified process is less error-prone. In addition, our Matlab program extracted the output factors faithfully from data logging, and the discrepancy between the automatic and manual measurement is within ±0.3%. For two separate automated measurements 30 days apart, consistency check shows a discrepancy within ±1% for 6MV photon with a 60 degree wedge. Conclusion: Automated output factor measurements can save time by 40% when compared with conventional manual approach. This work laid ground for further improvement for the automation of Linac commissioning.

43 PARTICLE ACCELERATORS↗

DOE OSTI2024

Impact of Color Space and Color Resolution on Vehicle Recognition Models

In this study, we analyze both linear and nonlinear color mappings by training on versions of a curated dataset collected in a controlled campus environment. We experiment with color space and color resolution to assess model performance in vehicle recognition tasks. Color encodings can be designed in principle to highlight certain vehicle characteristics or compensate for lighting differences when assessing potential matches to previously encountered objects. The dataset used in this work includes imagery gathered under diverse environmental conditions, including daytime and nighttime lighting. Experimental results inform expectations for possible improvements with automatic color space selection through feature learning. Moreover, we find there is only a gradual decrease in model performance with degraded color resolution, which suggests the need for simplified data collection and processing. By focusing on the most critical features, we could see improved model generalization and robustness, as the model becomes less prone to overfitting to noise or irrelevant details in the data. Such a reduction in resolution will lower computational complexity, leading to quicker training and inference times.

47 OTHER INSTRUMENTATION↗

DOE OSTI2022

SNAD transient miner: Finding missed transient events in ZTF DR4 using k-D trees

Here we report the automatic detection of 11 transients (7 possible supernovae and 4 active galactic nuclei candidates) within the Zwicky Transient Facility fourth data release (ZTF DR4), all of them observed in 2018 and absent from public catalogs. Among these, three were not part of the ZTF alert stream. Our transient mining strategy employs 41 physically motivated features extracted from both real light curves and four simulated light curve models (SN Ia, SN II, TDE, SLSN-I). These features are input to a k-D tree algorithm, from which we calculate the 15 nearest neighbors. After pre-processing and selection cuts, our dataset contained approximately a million objects among which we visually inspected the 105 closest neighbors from seven of our brightest, most well-sampled simulations, comprising 89 unique ZTF DR4 sources. Our result illustrates the potential of coherently incorporating domain knowledge and automatic learning algorithms, which is one of the guiding principles directing the SNAD team. It also demonstrates that the ZTF DR is a suitable testing ground for data mining algorithms aiming to prepare for the next generation of astronomical data.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

BFSVBF (BatFinder Smart Video BioFilter) [SWR-22-87] and Multi-class BatFinder Smart Video BioFilter Keras

Bats are notoriously difficult to study, therefore, identifying specific behavioral trends and the precise environmental conditions at the time of collision requires a monitoring solution that can reliably collect relevant data. To date, thermal infrared video surveillance has been extensively applied to study bats and has proven to be a powerful yet cumbersome tool. Current analytical approaches are time consuming because data processing data has not been fully automated. In the past, steps have been taken to record avian and bat activity in conjunction with complicated image processing techniques that separate species from other moving objects within the field of view (i.e. clouds and portions of the wind turbine). Once the videos are collected, the post-processing does not allow real time monitoring and identification, leading to a delay in both studying the behavior of these species and determining the effectiveness of any impact reduction strategy being studied. Moreover, object identification capability is lacking, thus limiting the usefulness of video data. To resolve these issues, we are using open source computer vision and machine learning techniques allowing for automatic detection of objects in real-time with the ability to correlate these objects with environmental variables and recording the flight paths of each object. The code has gone through five rounds of development with images used to train the models. This advancement allows for automated real-time data collection, identification, and tracking, thereby eliminating the need for long and tedious post-analysis processing of the videos. We will discuss the two open source and publicly available machine learning models developed within this scope of this work: 1) a binary model with a 97.5% accuracy in identifying the difference between an object and an empty scene, including wind turbine and clouds; and 2) a multiple classification model with the capability of identifying the type of object detected: bats (90% accuracy), birds (83% accuracy), insects (69% accuracy) and non-biological (99% accuracy).

Yarbrough, John↗

DOE OSTI2020

Removing imaging systematics from galaxy clustering measurements with Obiwan: application to the SDSS-IV extended Baryon Oscillation Spectroscopic Survey emission-line galaxy sample

This article presents the application of a new tool, Obiwan, which uses image simulations to determine the selection function of a galaxy redshift survey and calculate three-dimensional (3D) clustering statistics. Obiwan relies on a forward model of the process by which images of the night sky are transformed into a 3D large-scale structure catalogue, and offers several advantages over more traditional map-based techniques – such as operating on individual exposures and adopting a maximum likelihood approach. The photometric pipeline automatically detects and models galaxies and then generates a catalogue of such galaxies with detailed information for each one of them, including their location, redshift, and so on. Systematic biases in the imaging data are therefore imparted into the catalogues and must be accounted for in any scientific analysis of their information content. Obiwan simulates this process for samples selected from the Legacy Surveys imaging data. This imaging data will be used to select target samples for the next-generation Dark Energy Spectroscopic Instrument (DESI) experiment. Here, we apply Obiwan to a portion of the SDSS-IV extended Baryon Oscillation Spectroscopic Survey emission-line galaxies (ELGs). Systematic biases in the data are clearly identified and removed. We compare the 3D clustering results to those obtained by the map-based approach applied to the complete eBOSS Data Release 16 (DR16) sample. We find the results are consistent, thereby validating the eBOSS DR16 ELG catalogues, which is used to obtain cosmological results.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

A meta-learning based distribution system load forecasting model selection framework

This paper presents a meta-learning based, automatic distribution system load forecasting model selection framework. Furthermore, the framework includes the following processes: feature extraction, candidate model preparation and labeling, offline training, and online model recommendation. Using load forecasting needs and data characteristics as input features, multiple metalearners are used to rank the candidate load forecast models based on their forecasting accuracy. Then, a scoring-voting mechanism is proposed to weights recommendations from each meta-leaner and make the final recommendations. Heterogeneous load forecasting tasks with different temporal and technical requirements at different load aggregation levels are set up to train, validate, and test the performance of the proposed framework. Simulation results demonstrate that the performance of the meta-learning based approach is satisfactory in both seen and unseen forecasting tasks.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2016

SU-F-J-116: Clinical Experience-Based Verification and Improvement of a 4DCT Program

Purpose: To demonstrate the role of continuous improvement fulfilled by the Medical Physicist in clinical 4DCT and CBCT scanning. Methods: Lung (SABR and Standard) patients’ 4D respiratory motion and image data were reviewed over a 3, 6 and 12 month period following commissioning testing. By identifying trends of clinically relevant parameters and respiratory motions, variables were tested with a programmable motion phantom and assessed. Patient traces were imported to a motion phantom and 4DCT and CBCT imaging were performed. Cos6 surrogate and sup-inf motion was also programmed into the phantom to simulate the long exhale of patients for image contrast tests. Results: Patient surrogate motion amplitudes were 9.9+5.2mm (3–35) at 18+6bpm (6–30). Expiration/Inspiration time ratios of 1.4+0.5second (0.6–2.9) showed image contrast effects evident in the AveCT and 3DCBCT images. Small differences were found for patients with multiple 4DCT data sets. Patient motion assessments were simulated and verified with the phantom within 2mm. Initial image reviews to check for reconstructed artefacts and data loss identified a small number of patients with irregularities in the automatic placement of inspiration and expiration points. Conclusion: The Physicist’s involvement in the continuous improvements of a clinically commissioned technique, processes and workflows continues beyond the commissioning stage of a project. Our experience with our clinical 4DCT program shows that Physics presence is required at the clinical 4DCT scan to assist with technical aspects of the scan and also for clinical image quality assessment prior to voluming. The results of this work enabled the sharing of information from the Medical Physics group with the Radiation Oncologists and Radiation Therapists. This results in an improved awareness of clinical patient respiration variables and how they may affect 4D simulation images and also may also affect the treatment verification images.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2022

Image-domain DAS 3D VSP elastic transmission tomography

SUMMARY Full-wavefield elastic imaging of active-source seismic data acquired by downhole receivers commonly offers higher-resolution subsurface images in the vicinity of a borehole compared to conventional surface seismic data sets, which can lack higher-frequency wavefield components due to longer travel paths and increased attenuation. An increasingly used approach for downhole acquisition is vertical seismic profiling (VSP), which has become more attractive when coupled with distributed acoustic sensing (DAS) using optical fibres installed in wells. The main difficulty for generating high-quality images with full-wavefield imaging tools for DAS VSP data, though, is the need for an accurate velocity model. To build plausible velocity models using active-source DAS VSP data, we adopt a 3-D image-domain elastic transmission tomography technique, originally developed for surface-recorded passive (microseismic) data, by exchanging the source and receiver positions (i.e. reciprocity) to mimic a passive-seismic surface monitoring scenario. The inversion approach exploits various images for each source constructed through time-reverse imaging (TRI) of downgoing P- and S-wave first-arrival waveforms. The TRI process uses the kinetic term of the (extended) PS energy imaging condition that exhibits sufficient sensitivity to velocity model errors. The method automatically updates the P- and S-wave velocity models to optimize image focusing via adjoint-state inversion. We illustrate the efficacy of the adopted elastic inversion technique using an active-source DAS 3-D VSP field data set acquired in the North Slope of Alaska. The numerical experiments demonstrate that the inverted elastic velocity models can be further used in full-wavefield acoustic/elastic imaging algorithms to obtain accurate subsurface images.

Oren, Can (ORCID:000000019534265X)↗

DOE OSTI2025

Event Log / Raw Data

The WFIP3 event log is a curated record spanning 578 days of meteorological phenomena and field observations that complements the campaign’s high-frequency measurements. The log combines manually documented daily weather discussions with automatically derived indicators of key atmospheric processes, providing standardized, publicly available context to support model evaluation, forecast verification, and case-study selection for offshore boundary-layer research.

17 WIND ENERGY↗

DOE OSTI2022

Learning functional priors and posteriors from data and physics

In this work, we develop a new Bayesian framework based on deep neural networks to be able to extrapolate in space-time using historical data and to quantify uncertainties arising from both noisy and gappy data in physical problems. Specifically, the proposed approach has two stages: (1) prior learning and (2) posterior estimation. At the first stage, we employ the physics-informed Generative Adversarial Networks (PI-GAN) to learn a functional prior either from a prescribed function distribution, e.g., Gaussian process, or from historical data and physics. At the second stage, we employ the Hamiltonian Monte Carlo (HMC) method to estimate the posterior in the latent space of PI-GANs. In addition, we use two different approaches to encode the physics: (1) automatic differentiation, used in the physicsinformed neural networks (PINNs) for scenarios with explicitly known partial differential equations (PDEs), and (2) operator regression using the deep operator network (DeepONet) for PDE-agnostic scenarios. We then test the proposed method for (1) meta-learning for one-dimensional regression, and forward/inverse PDE problems (combined with PINNs); (2) PDE-agnostic physical problems (combined with DeepONet), e.g., fractional diffusion as well as saturated stochastic (100-dimensional) flows in heterogeneous porous media; and (3) spatial-temporal regression problems, i.e., inference of a marine riser displacement field using experimental data from the Norwegian Deepwater Programme (NDP). The results demonstrate that the proposed approach can provide accurate predictions as well as uncertainty quantification given very limited scattered and noisy data, since historical data could be available to provide informative priors. In summary, the proposed method is capable of learning flexible functional priors, e.g., both Gaussian and non-Gaussian process, and can be readily extended to big data problems by enabling mini-batch training using stochastic HMC or normalizing flows since the latent space is generally characterized as low dimensional.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2019

Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities

Buildings account for 35% of the global final energy demand. Efficiency improvements and advanced control strategies have a significant impact in the reduction of energy costs and CO2 emissions. Building energy simulation is widely used to help planners, contractors, and building owners analyse diverse options regarding the planning and management of energy consumption in buildings. Furthermore, recent advances in data processing and computing have led to the development of sophisticated machine learning algorithms that can learn from large datasets, e.g., sensor data from buildings, and use them to develop building-specific adaptive and automatic energy controllers. Control algorithms, such as deep reinforcement learning can tune themselves, are model-free, and economical to implement. In this paper, we introduce an integrated simulation environment that combines CitySim, a fast building energy simulator, and TensorFlow, a platform for efficient implementation of advanced machine learning algorithms. The integration is achieved via Keras - an API for TensorFlow - and a set of text and csv files for data transfer between the applications. This new environment will allow researchers to investigate novel learning control algorithms, and demonstrate their robustness and potential for diverse applications in the built environment. We present two case studies for energy savings and demand response, respectively.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2026

NeuNorm

NeuNorm is a scipp-based Python library for neutron imaging normalization and time-of-flight (TOF) data processing at Oak Ridge National Laboratory imaging facilities (MARS at HFIR and VENUS at SNS). NeuNorm 2.0 is a complete, scipp-based rewrite of the original NeuNorm normalization library, adding HDF5 output, automatic uncertainty propagation, and TOF/event-mode processing.

Zhang, Chen [Oak Ridge National Laboratory (ORNL),↗

DOE OSTI2025

Physics-informed latent neural operator for real-time predictions of time-dependent parametric PDEs

Deep operator network (DeepONet) has shown significant promise as surrogate models for systems governed by partial differential equations (PDEs), enabling accurate mappings between infinite-dimensional function spaces. However, when applied to systems with high-dimensional input-output mappings arising from large numbers of spatial and temporal collocation points, these models often require heavily overparameterized networks, leading to long training times. Latent DeepONet addresses some of these challenges by introducing a two-step approach: first learning a reduced latent space using a separate model, followed by operator learning within this latent space. While efficient, this method is inherently data-driven and lacks mechanisms for incorporating physical laws, limiting its robustness and generalizability in data-scarce settings. Here, in this work, we propose PI-Latent-NO, a physics-informed latent neural operator framework that integrates governing physics directly into the learning process. Our architecture features two coupled DeepONets trained end-to-end: a Latent-DeepONet that learns a low-dimensional representation of the solution, and a Reconstruction-DeepONet that maps this latent representation back to the physical space. By embedding PDE constraints into the training via automatic differentiation, our method eliminates the need for labeled training data and ensures physics-consistent predictions. The proposed framework is both memory and compute-efficient, exhibiting near-constant scaling with problem size and demonstrating significant speedups over traditional physics-informed operator models. We validate our approach on a range of parametric PDEs, showcasing its accuracy, scalability, and suitability for real-time prediction in complex physical systems.

Latent representations↗

DOE OSTI2016

iTOUGH2: A multiphysics simulation-optimization framework for analyzing subsurface systems

We report iTOUGH2 is a simulation-optimization framework for the TOUGH suite of nonisothermal multiphase flow models and related simulators of geophysical, geochemical, and geomechanical processes. After appropriate parameterization of subsurface structures and their properties, iTOUGH2 runs simulations for multiple parameter sets and analyzes the resulting output for parameter estimation through automatic model calibration, local and global sensitivity analyses, data-worth analyses, and uncertainty propagation analyses. Development of iTOUGH2 is driven by scientific challenges and user needs, with new capabilities continually added to both the forward simulator and the optimization framework. This paper provides a summary description of methods and features implemented in iTOUGH2, and discusses the usefulness and limitations of an integrated simulation-optimization workflow in support of the characterization and analysis of complex multiphysics subsurface systems.

58 GEOSCIENCES↗