Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2011

RACORO aerosol data processing

The RACORO aerosol data (cloud condensation nuclei (CCN), condensation nuclei (CN) and aerosol size distributions) need further processing to be useful for model evaluation (e.g., GCM droplet nucleation parameterizations) and other investigations. These tasks include: (1) Identification and flagging of 'splash' contaminated Twin Otter aerosol data. (2) Calculation of actual supersaturation (SS) values in the two CCN columns flown on the Twin Otter. (3) Interpolation of CCN spectra from SGP and Twin Otter to 0.2% SS. (4) Process data for spatial variability studies. (5) Provide calculated light scattering from measured aerosol size distributions. Below we first briefly describe the measurements and then describe the results of several data processing tasks that which have been completed, paving the way for the scientific analyses for which the campaign was designed. The end result of this research will be several aerosol data sets which can be used to achieve some of the goals of the RACORO mission including the enhanced understanding of cloud-aerosol interactions and improved cloud simulations in climate models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

HERO WEC V1 Upgrade - 2023 Laboratory Testing (processed data)

The following submission includes processed laboratory data from NREL's Hydraulic and Electric Reverse Osmosis Wave Energy Converter (HERO WEC), in the form of MATLAB workspaces. This dataset was created using NREL's Large Amplitude Motion Platform (LAMP) and collected between August and September 2023. Included with this submission is a test log of all the processed data "HERO WEC LAMP test run log.xlsx" so that the user can easily find the data of interest. Additionally, more detailed descriptions of the type of data and how it was processed, or calculated, can be found in the document titled "Lamp Data Description.docx". The MATLAB workspaces can be visualized using the file "LAMP_Data_Viewer_Ver2.m/mlx". The user simply needs to upload the workspace of interest and run the file "LAMP_Data_Viewer_Ver2.m/mlx". Both the .m and .mlx file format has been provided depending on the user's preference. The MATLAB workspaces have been separated into zip files corresponding to either Drivetrain, Hydraulic, or Electric configuration runs representing the respective test cases that were run. The drivetrain runs were used to characterize the drivetrain only (no pump or generator). The Hydraulic runs represent the configuration when the seawater pump is installed, and the Electric runs represents the configuration when the generator is installed. The following sub-categories of data are included for each type: - DW - Deep water sine wave profile (not run in drivetrain configuration) - Heave - Heave only sine wave profile - Heave_NoRO (hydraulic configuration only) - Heave_ACC (hydraulic configuration only) - IR - Surge and heave irregular wave profile (not run in drivetrain configuration) - RW - Heave only profile created from real world encoder data (not run in drivetrain configuration) For those interested in the raw, unprocessed, data the authors have created a separate submission, linked below. This submission includes the raw TDMS files and associated files necessary to translate the data into either python or MATLAB formats. This data set has been developed by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided U.S. Department of Energy Office of Energy Efficiency and Renewable Energy Water Power Technologies Office.

16 TIDAL AND WAVE POWER↗
DOE OSTI2016

A new data processing technique for Rayleigh-Taylor instability growth experiments

Typical face-on experiments for Rayleigh-Taylor instability study involve the time-resolved radiography of an accelerated foil with line-of-sight of the radiography along the direction of motion. The usual method which derives perturbation amplitudes from the face-on images reverses the actual image transmission procedure, so the obtained results will have a large error in the case of large optical depth. In order to improve the accuracy of data processing, a new data processing technique has been developed to process the face-on images. This technique based on convolution theorem, refined solutions of optical depth can be achieved by solving equations. Furthermore, we discuss both techniques for image processing, including the influence of modulation transfer function of imaging system and the backlighter spatial profile. Besides, we use the two methods to the process the experimental results in Shenguang-II laser facility and the comparison shows that the new method effectively improve the accuracy of data processing.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2020

Big Data Processing for Power Grid Event Detection

In this paper we present the application of big data processing for the development of machine learning (ML) models to detect relevant events in power grid operations. This is based on almost 20TB of phasor measurement unit data corresponding to up to two years of operation of three grid interconnections which provide power to most of the United States. A significant aspect of the work consists in having all data processing performed on a single standard GPU server, from pre-processing to ML model training and testing. We describe the data and computational infrastructure, challenges faced and methods used in data processing, main findings and results. The ML approach employed for best utilization of the big data is also discussed, including sample results.

Paes Leao, Bruno↗
DOE OSTI2022

Data Processing Unit Services Module

The Data Processing Services Module (DPUSM) provides the ability to perform pluggable compression, erasure coding, checksuming and other important file system operations within the Linux kernel. The pluggable provider interface allows for the use of hardware acceleration of those services. In-kernel file systems are then able to use these functions to use these accelerators to perform operations that are normally run on the processor, resulting in improved file system performance. Third parties will register "providers" with the DPUSM to communicate with their respective accelerators. Providers will implement functions with DPUSM API signatures so that the DPUSM can translate the data inputted by users of the DPUSM into data that providers recognize.

Lee, Jason↗
DOE OSTI2024

DOE Buoy 130 Lidar / Processed Data

This dataset contains preliminary processed data from DOE Buoy 130 deployed for WFIP-3. A new ZX 300 M lidar is currently deployed on the buoy.

17 WIND ENERGY↗
DOE OSTI2024

WFIP3 Lidar / Processed Data

This dataset contains preliminary processed data from the Leosphere WindCube V2.1 deployed at Nantucket, MA for WFIP-3. The lidar was tall tower validated in May 2023.

17 WIND ENERGY↗
DOE OSTI2024

WFIP3 Lidar / Processed Data

This dataset contains standardized, processed data from the Leosphere WindCube V2.1 deployed at Nantucket, MA for WFIP-3. The lidar was tall tower validated in May 2023.

17 WIND ENERGY↗
DOE OSTI2025

Lidar / Processed Data

This dataset contains preliminary processed data from the WHOI ASIT deployed for WFIP-3. A ZX 300M is currently installed at the tower and has been deployed there since September 2021.

17 WIND ENERGY↗
DOE OSTI2017

The NJOY Nuclear Data Processing System, Version 2016

The NJOY Nuclear Data Processing System, version 2016, is a comprehensive computer code package for producing pointwise and multigroup cross sections and related quantities from evaluated nuclear data in the ENDF-4 through ENDF-6 legacy card-image formats. NJOY works with evaluated files for incident neutrons, photons, and charged particles, producing libraries for a wide variety of particle transport and reactor analysis codes.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

High-Throughput Data Processing at FRIB Using ESnet

Real-time or nearly real-time (nearline) data processing methods are critical tools as detector technologies and data acquisition (DAQ) systems allow for higher data rates and volumes. The introduction of the energy sciences network (ESnet), a U.S. Department of Energy (DOE) supported high-speed network for scientific research, creates opportunities to leverage the computing power of DOE facilities like the National Energy Research Scientific Computing Center (NERSC). As a first step toward realizing a DOE Office of Science Integrated Research Infrastructure (IRI) pattern, an automated workflow was developed to remotely process data obtained from a nuclear physics experiment at the Facility for Rare Isotope Beams (FRIB) at NERSC with data transferred between FRIB and NERSC over ESnet. The workflow demonstrated the ability to process one week’s worth of experimental data in approximately 90 min and was used successfully for nearline analysis during a recently completed FRIB experiment. Here, a summary of the workflow development and results of recent demonstrations will be presented.

Data processing↗
DOE OSTI2021

Data processing pipeline for Tianlai experiment

The Tianlai project is a 21cm intensity mapping experiment for detecting dark energy by measuring the baryon acoustic oscillation (BAO) features in the large scale structure power spectrum. This experiment provides an opportunity to test the data processing methods for cosmological 21cm signal extraction, which is still a great challenge in current radio astronomy research. The 21cm signal is much weaker than the foregrounds and easily aected by the imperfections in the instrumental responses. Furthermore, processing the large volumes of interferometer data poses a practical challenge. We have developed a data processing pipeline called tlpipe to process the drift scan survey data from the Tianlai experiment. It performs oine data processing tasks such as radio frequency interference (RFI) agging, array calibration, binning, and map-making, etc. It also includes utility functions needed for the data analysis, such as data selection, transformation, visualization and others. A number of new algorithms are implemented, for example the eigenvector decomposition method for array calibration and the Tikhnov regularization for m-mode analysis. In this paper we describe the design and implementation of the pipeline and illustrate its functions with some analysis of real data. Finally, we outline directions for future development of this publicly code.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2025

Real-time data processing for serial crystallography experiments

We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames ('hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.

47 OTHER INSTRUMENTATION↗
DOE OSTI2015

A Prescription for List-Mode Data Processing Conventions

There are a variety of algorithmic approaches available to process list-mode pulse streams to produce multiplicity histograms for subsequent analysis. In the development of the INCC v6.0 code to include the processing of this data format, we have noted inconsistencies in the “processed time” between the various approaches. The processed time, tp, is the time interval over which the recorded pulses are analyzed to construct multiplicity histograms. This is the time interval that is used to convert measured counts into count rates. The observed inconsistencies in tp impact the reported count rate information and the determination of the error-values associated with the derived singles, doubles, and triples counting rates. This issue is particularly important in low count-rate environments. In this report we will present a prescription for the processing of list-mode counting data that produces values that are both correct and consistent with traditional shift-register technologies. It is our objective to define conventions for list mode data processing to ensure that the results are physically valid and numerically aligned with the results from shift-register electronics.

List mode↗
DOE OSTI2014

Tightly integrated single- and multi-crystal data collection strategy calculation and parallelized data processing in JBluIce beamline control system

The calculation of single- and multi-crystal data collection strategies and a data processing pipeline have been tightly integrated into the macromolecular crystallographic data acquisition and beamline control software JBluIce . Both tasks employ wrapper scripts around existing crystallographic software. JBluIce executes scripts through a distributed resource management system to make efficient use of all available computing resources through parallel processing. The JBluIce single-crystal data collection strategy feature uses a choice of strategy programs to help users rank sample crystals and collect data. The strategy results can be conveniently exported to a data collection run. The JBluIce multi-crystal strategy feature calculates a collection strategy to optimize coverage of reciprocal space in cases where incomplete data are available from previous samples. The JBluIce data processing runs simultaneously with data collection using a choice of data reduction wrappers for integration and scaling of newly collected data, with an option for merging with pre-existing data. Data are processed separately if collected from multiple sites on a crystal or from multiple crystals, then scaled and merged. Results from all strategy and processing calculations are displayed in relevant tabs of JBluIce .

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2013

Data processing with microcode designed with source coding

Programming for a data processor to execute a data processing application is provided using microcode source code. The microcode source code is assembled to produce microcode that includes digital microcode instructions with which to signal the data processor to execute the data processing application.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Summary report of the 4th IAEA Technical Meeting on Fusion Data Processing, Validation and Analysis (FDPVA)

The objective of the Fourth Technical Meeting on Fusion Data Processing, Validation and Analysis was to provide a platform during which a set of topics relevant to fusion data processing, validation and analysis are discussed with the view of extrapolating needs to next step fusion devices such as ITER. The validation and analysis of experimental data obtained from diagnostics used to characterize fusion plasmas are crucial for a knowledge-based understanding of the physical processes governing the dynamics of these plasmas. This paper presents the recent progress and achievements in the domain of plasma diagnostics and synthetic diagnostics data analysis (including image processing, regression analysis, inverse problems, deep learning, machine learning, big data and physics-based models for control) reported at the meeting. The progress in these areas highlight trends observed in current major fusion confinement devices. A special focus is dedicated on data analysis requirements for ITER and DEMO with a particular attention paid to artificial intelligence for automatization and improving reliability of control processes.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2023

TomocuPy – efficient GPU-based tomographic reconstruction with asynchronous data processing

Fast 3D data analysis and steering of a tomographic experiment by changing environmental conditions or acquisition parameters require fast, close to real-time, 3D reconstruction of large data volumes. Here a performance-optimized TomocuPy package is presented as a GPU alternative to the commonly used central processing unit (CPU) based TomoPy package for tomographic reconstruction. TomocuPy utilizes modern hardware capabilities to organize a 3D asynchronous reconstruction involving parallel read/write operations with storage drives, CPU–GPU data transfers, and GPU computations. In the asynchronous reconstruction, all the operations are timely overlapped to almost fully hide all data management time. Since most cameras work with less than 16-bit digital output, the memory usage and processing speed are furthermore optimized by using 16-bit floating-point arithmetic. As a result, 3D reconstruction with TomocuPy became 20–30 times faster than its multi-threaded CPU equivalent. Full reconstruction (including read/write operations and methods initialization) of a 2048 3 tomographic volume takes less than 7 s on a single Nvidia Tesla A100 and PCIe 4.0 NVMe SSD, and scales almost linearly increasing the data size. To simplify operation at synchrotron beamlines, TomocuPy provides an easy-to-use command-line interface. Efficacy of the package was demonstrated during a tomographic experiment on gas-hydrate formation in porous samples, where a steering option was implemented as a lens-changing mechanism for zooming to regions of interest.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗