Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Intelligent experiments through real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and future EIC detectors

This R&D project, initiated by the DOE Nuclear Physics AI-Machine Learning initiative in 2022, leverages AI to address data processing challenges in high-energy nuclear experiments (RHIC, LHC, and future EIC). Our focus is on developing a demonstrator for real-time processing of high-rate data streams from sPHENIX experiment tracking detectors. The limitations of a 15 kHz maximum trigger rate imposed by the calorimeters can be negated by intelligent use of streaming technology in the tracking system. The approach efficiently identifies low momentum rare heavy flavor events in high-rate p+p collisions (3MHz), using Graph Neural Network (GNN) and High Level Synthesis for Machine Learning (hls4ml). Success at sPHENIX promises immediate benefits, minimizing resources and accelerating the heavy-flavor measurements. The approach is transferable to other fields. For the EIC, we develop a DIS-electron tagger using Artificial Intelligence - Machine Learning (AI-ML) algorithms for real-time identification, showcasing the transformative potential of AI and FPGA technologies in high-energy nuclear and particle experiments real-time data processing pipelines.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2015

Adaptive Data Processing Technique for Lidar-Assisted Control to Bridge the Gap between Lidar Systems and Wind Turbines: Preprint

This paper presents first steps toward an adaptive lidar data processing technique crucial for lidar-assisted control in wind turbines. The prediction time and the quality of the wind preview from lidar measurements depend on several factors and are not constant. If the data processing is not continually adjusted, the benefit of lidar-assisted control cannot be fully exploited, or can even result in harmful control action. An online analysis of the lidar and turbine data are necessary to continually reassess the prediction time and lidar data quality. In this work, a structured process to develop an analysis tool for the prediction time and a new hardware setup for lidar-assisted control are presented. The tool consists of an online estimation of the rotor effective wind speed from lidar and turbine data and the implementation of an online cross correlation to determine the time shift between both signals. Further, initial results from an ongoing campaign in which this system was employed for providing lidar preview for feed-forward pitch control are presented.

17 WIND ENERGY↗
DOE OSTI2026

NLR Data Processing Pipeline for MADIS [SWR-26-050]

The NLR Data Processing Pipeline for MADIS software package is for downloading, processing, and performing QA/QC on MADIS data. Designed to handle the following steps: 1) Download all MADIS data as compressed netcdf files for a given time period. 2) Unpack netcdf files into timeseries csvs for each coordinate within the given bounding box. 3) Process the csvs to filter according to quality control checks and convert variables to correct units. 4) Write processed csvs to a single nc file.

Benton, Brandon [National Laboratory of the Rockie↗
DOE OSTI2013

Evaluation of the flow-based programming (FBP) paradigm as an alternative to standard programming practices in physics data processing applications

The majority of developed physics data processing applications (PDP) are single, sequential processes that start at a point in time, and advance one step at a time until they are finished. In the current era of cloud computing and multi-core hardware architectures this approach has noticeable limitations. In this paper we present a detailed evaluation of the FBP-based Clas12 event reconstruction program that was deployed and operated both in cloud and in batch processing environments. We demonstrate the programming methodology and discuss some of the issues and optimizations affecting performance. We will also discuss our choice of using the Petri-Net process modeling formalism for the representation of the Clas12 PDP application building blocks which exhibit concurrency, parallelism, and synchronization.

Abbott, David↗
DOE OSTI2013

Nuclear data processing for energy release and deposition calculations in the MC21 Monte Carlo code

With the recent emphasis in performing multiphysics calculations using Monte Carlo transport codes such as MC21, the need for accurate estimates of the energy deposition-and the subsequent heating - has increased. However, the availability and quality of data necessary to enable accurate neutron and photon energy deposition calculations can be an issue. A comprehensive method for handling the nuclear data required for energy deposition calculations in MC21 has been developed using the NDEX nuclear data processing system and leveraging the capabilities of NJOY. The method provides a collection of data to the MC21 Monte Carlo code supporting the computation of a wide variety of energy release and deposition tallies while also allowing calculations with different levels of fidelity to be performed. Detailed discussions on the usage of the various components of the energy release data are provided to demonstrate novel methods in borrowing photon production data, correcting for negative energy release quantities, and adjusting Q values when necessary to preserve energy balance. Since energy deposition within a reactor is a result of both neutron and photon interactions with materials, a discussion on the photon energy deposition data processing is also provided. (authors)

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Tidal Energy Resource Characterization, Velocity and Turbulence Measurements, Processed Data, Cook Inlet, AK, 2021

This submission contains processed datasets from a long-term deployment of 3 moorings and a transect survey of the proposed tidal energy site off the East Forelands in Cook Inlet, AK. The long-term mooring datasets were created from 8 instruments mounted on a Terrasond High Energy Oceanographic Mooring (THEOM) bottom lander and two Mid-Water Mooring (MWM) Stablemoor buoys from 1 July 2021 to 31 August 2021 (60 days). The west-most mooring (MWM1) was deployed at 60.720225 N, 151.436196 W in ~50 m of water. The middle mooring (THEOM) was deployed at 60.720703 N, 151.429500 W in ~52 m of water. The east-most buoy (MWM2) was deployed at 60.720081 N, 151.420896 W in ~50 m of water. Each Stablemoor carried three instruments: 1. A Nortek Vector acoustic Doppler velocimeter (ADV) mounted at the Stablemoor's nose. Data were recorded at 8 Hz on a 5 minute duty cycle every 20 minutes. Data was motion-corrected using the internal IMU and external ADCP bottom-track data and then bin-averaged into 4 minute bins and converted to the Principal (streamwise, cross-stream, vertical) coordinate system. (Note: 30 seconds were trimmed from the beginning and end of each 5 minute duty cycle to account for the filter end-effects from turning on and turning off the IMU.) 2. A down-looking Nortek Signature 1000 kHz acoustic Doppler current profiler (ADCP) mounted in the first Stablemoor instrument well. Data were recorded in 2 Hz with 5-beam burst and bottom-track enabled. Processed data has been averaged into 10 minute bins and converted into the Principal coordinate system. 3. An up-looking Nortek Signature 1000 kHz acoustic Doppler current profiler (ADCP) mounted in the second Stablemoor instrument well. Data were recorded at 4 Hz with 5 beam burst enabled. Processed data has been averaged into 10 minute bins and converted into the Principal coordinate system. Note: the down-facing ADCP on MWM1 failed on July 10th, 2021, only recording 9 days of data. Because ADV motion-correction required bottom track, the ADV from MWM1 also only has 9 days processed. Additionally, only 25 days of data were processed from the MWM2 ADV because it appeared to have been impacted by debris on 7/25. Two instruments were mounted on the THEOM (see MHKDR link further below for THEOM raw data): 4. A Nortek Vector acoustic Doppler velocimeter (ADV). Data were recorded at 8 Hz on a 5 minute duty cycle every 20 minutes. Data was bin-averaged into 5 minute bins, and converted to the Principal coordinate system. 5. A Nortek Signature 500 kHz acoustic Doppler current profiler (ADCP). Data were recorded in 4 Hz in the beam coordinate system from all 5 beams. Processed data has been averaged into 10 minutes bins and converted to the Principal coordinate system.

16 TIDAL AND WAVE POWER↗
DOE OSTI2020

Planck 2018 Results - II. Low Frequency Instrument data processing

We present a final description of the data-processing pipeline for the Planck Low Frequency Instrument (LFI), implemented for the 2018 data release. Several improvements have been made with respect to the previous release, especially in the calibration process and in the correction of instrumental features such as the effects of nonlinearity in the response of the analogue-to-digital converters. We provide a brief pedagogical introduction to the complete pipeline, as well as a detailed description of the important changes implemented. Self-consistency of the pipeline is demonstrated using dedicated simulations and null tests. We present the final version of the LFI full sky maps at 30, 44, and 70 GHz, both in temperature and polarization, together with a refined estimate of the solar dipole and a final assessment of the main LFI instrumental parameters.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

PvaPy streaming framework for real-time data processing

User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.

EPICS pvAccess↗
DOE OSTI2023

Demonstrating Cross-Facility Data Processing at Scale with Laue Microdiffraction

In February and April 2023 live, at-scale data processing demonstrations were conducted between the Advanced Photon Source (APS), a synchrotron light source, and the Argonne Leadership Computing Facility (ALCF). These tests were run as part of a novel beamline technique: coded aperture laue micro-diffraction. This technique requires a significant amount of compute to decode appeture patterns embedded in the detector stream. An autonomous system was able to send data to ALCF during an experiment, utilize 50 nodes of the Polaris supercomputer to process 6-12 hour scans, and return the data back to the APS within 12-15 minutes behind the detector. With scan points arriving every 72 seconds, the system kept up with the beamline, potentially enabling in-experiment analysis. The data processing system utilizes Globus infrastructure and an on-demand queue to dynamically acquire nodes on Polaris. The underlying reconstruction algorithms were parallelized via MPI and accelerated with custom CUDA kernels.

Prince, Michael↗
DOE OSTI2015

Laser scanner data processing and 3D modeling using a free and open source software

The laser scanning is a technology that allows in a short time to run the relief geometric objects with a high level of detail and completeness, based on the signal emitted by the laser and the corresponding return signal. When the incident laser radiation hits the object to detect, then the radiation is reflected. The purpose is to build a three-dimensional digital model that allows to reconstruct the reality of the object and to conduct studies regarding the design, restoration and/or conservation. When the laser scanner is equipped with a digital camera, the result of the measurement process is a set of points in XYZ coordinates showing a high density and accuracy with radiometric and RGB tones. In this case, the set of measured points is called “point cloud” and allows the reconstruction of the Digital Surface Model. Even the post-processing is usually performed by closed source software, which is characterized by Copyright restricting the free use, free and open source software can increase the performance by far. Indeed, this latter can be freely used providing the possibility to display and even custom the source code. The experience started at the Faculty of Engineering in Catania is aimed at finding a valuable free and open source tool, MeshLab (Italian Software for data processing), to be compared with a reference closed source software for data processing, i.e. RapidForm. In this work, we compare the results obtained with MeshLab and Rapidform through the planning of the survey and the acquisition of the point cloud of a morphologically complex statue.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2025

Phlex: Parallel, Hierarchical, and Layered EXecution of data-processing algorithms

Phlex is a computing framework supporting the parallel, hierarchical, and layered execution of data-processing algorithms. It is based on the functional-programming paradigm, thus guaranteeing thread-safety when invoking user-defined pure functions. Phlex allows users to specify arbitrary graph-based hierarchies of data organization, enabling more flexible processing of data as required by the constraints of the program.

Knoepfel, KyleJ. [Fermi National Accelerator Labor↗
DOE OSTI2023

Lidar / Processed Data

Wind energy research Note that the processed lidar data has the same metadata as the processed buoy data. These were once considered one dataset, but now the lidar data has been separated to form its own dataset.

17 WIND ENERGY↗
DOE OSTI2015

AN ATLAS OF CORONAL ELECTRON DENSITY AT 5R{sub ⊙}. I. DATA PROCESSING AND CALIBRATION

Tomography of the solar corona can provide cruicial constraints for models of the low corona, unique information on changes in coronal structure and rotation rates, and a valuable boundary condition for models of the heliospheric solar wind. This is the first of a series of three papers which aim to create a set of maps of the coronal density over an extended period (1996-present). The papers will describe the data processing and calibration (this paper), the tomography method (Paper II), and the resulting atlas of coronal electron density at a height of 5 R{sub ⊙} between years 1996–2014 (Paper III). This first paper presents a detailed description of data processing and calibration for the Large-Angle and Spectrometric Coronagraph (LASCO) C2 instrument on board the Solar and Heliospheric Observatory (SOHO) and the COR2 instruments of the Sun Earth Connection Coronal and Heliospheric Investigation (SECCHI) package on board the Solar Terrestial Relations Observatory (STEREO) A and B spacecraft. The methodology includes noise suppression, background subtraction, separation of large dynamic events, conversion of total brightness to K-coronal brightness, and simple functions for cross-calibration between C2/LASCO and COR2/SECCHI. Comparison of the brightness of stars between LASCO C2 total and polarized brightness (pB) observations provide in-flight calibration factors for the pB observations, resulting in considerable improved agreement between C2 and COR2 A, and elimination of curious artifacts in the C2 pB images. The cross-calibration between LASCO C2 and the STEREO coronagraphs allows, for the first time, the potential use of multi-spacecraft coronagraph data for tomography and for coronal mass ejection analysis.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers

The prompt reconstruction of the data recorded from the Large Hadron Collider (LHC) detectors has always been addressed by dedicated resources at the CERN Tier-0. Such workloads come in spikes due to the nature of the operation of the accelerator and in special high load occasions experiments have commissioned methods to distribute (spill-over) a fraction of the load to sites outside CERN. The present work demonstrates a new way of supporting the Tier-0 environment by provisioning resources elastically for such spilled-over workflows onto the Piz Daint Supercomputer at CSCS. Furthermore, this is implemented using containers, tuning the existing batch scheduler and reinforcing the scratch file system, while still using standard Grid middleware. ATLAS, CMS and CSCS have jointly run selected prompt data reconstruction on up to several thousand cores on Piz Daint into a shared environment, thereby probing the viability of the CSCS high performance computer site as on demand extension of the CERN Tier-0, which could play a role in addressing the future LHC computing challenges for the high luminosity LHC.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

New and Extended Data Processing of Mars Odyssey Neutron Spectrometer Data

The Los Alamos National Laboratory designed and built Mars Odyssey Neutron Spectrometer (MONS) has been operating and collecting data from February 2002 to the present. MONS measures the neutron leakage albedo from galactic cosmic ray bombardment of Mars. These signals can indicate the presence of near-surface water deposits on Mars, and can also be used to study properties of the seasonal polar CO 2 ice caps. This work outlines a new analysis of the MONS data that results in new and extended time-series maps of MONS thermal and epithermal neutron data. The new data are compared to previous publications on the MONS instrument. We then present preliminary results studying the inter-annual variability in the polar regions of Mars based on 8 Mars-Years of MONS data from the new dataset.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2011

User's Guide to Pre-Processing Data in Universal Translator 2 for the Energy Charting and Metrics Tool (ECAM)

This document is a user's guide for the Energy Charting and Metrics Tool to facilitate the examination of energy information from buildings, reducing the time spent analyzing trend and utility meter data. This user guide was generated to help pre-process data with the intention of utilizing the Energy Charting and Metrics (ECAM) tool to improve building operational efficiency. There are numerous occasions when the metered data that is received from the building automation system (BAS) isn't in the right format acceptable for ECAM. This includes, but isn't limited to, cases such as inconsistent time-stamps for the trends (e.g., each trend has its own time-stamp), data with holes (e.g., some time-stamps have data and others are missing data), each point in the BAS is trended and exported into an individual .csv or .txt file, the time-stamp is unrecognizable by ECAM, etc. After reading through this user guide, the user should be able to pre-process all data files and be ready to use this data in ECAM to improve their building operational efficiency.

99 GENERAL AND MISCELLANEOUS↗