Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Streaming Data from Experimental Facilities to Supercomputers for Real-Time Data Processing

In this paper we demonstrate direct data streaming from instruments and detectors at a large-scale experimental facility to a supercomputer for real-time data processing and feedback. Streaming data to supercomputers introduces the potential for novel scientific applications and workflow models, including the ability to provide real-time feedback from very large datasets during an experiment and the integration of real-time ML training and inference at scale. We discuss a successful demonstration for real-time processing of data from the Advanced Photon Source (APS) on the Polaris supercomputer using an EPICS-based streaming framework. We describe the capabilities of the streaming framework itself, and outline the architecture that allows us to process experimentally derived data on a supercomputer without file-based data transfers. We present throughput measurements that are indicative of system performance capable of sustaining the expected data production rates of the facility, as well as discuss some outstanding challenges and our future directions.

real-time processing↗
DOE OSTI2024

HERO WEC V1 Upgrade - 2023 Laboratory Testing (processed data)

The following submission includes processed laboratory data from NREL's Hydraulic and Electric Reverse Osmosis Wave Energy Converter (HERO WEC), in the form of MATLAB workspaces. This dataset was created using NREL's Large Amplitude Motion Platform (LAMP) and collected between August and September 2023. Included with this submission is a test log of all the processed data "HERO WEC LAMP test run log.xlsx" so that the user can easily find the data of interest. Additionally, more detailed descriptions of the type of data and how it was processed, or calculated, can be found in the document titled "Lamp Data Description.docx". The MATLAB workspaces can be visualized using the file "LAMP_Data_Viewer_Ver2.m/mlx". The user simply needs to upload the workspace of interest and run the file "LAMP_Data_Viewer_Ver2.m/mlx". Both the .m and .mlx file format has been provided depending on the user's preference. The MATLAB workspaces have been separated into zip files corresponding to either Drivetrain, Hydraulic, or Electric configuration runs representing the respective test cases that were run. The drivetrain runs were used to characterize the drivetrain only (no pump or generator). The Hydraulic runs represent the configuration when the seawater pump is installed, and the Electric runs represents the configuration when the generator is installed. The following sub-categories of data are included for each type: - DW - Deep water sine wave profile (not run in drivetrain configuration) - Heave - Heave only sine wave profile - Heave_NoRO (hydraulic configuration only) - Heave_ACC (hydraulic configuration only) - IR - Surge and heave irregular wave profile (not run in drivetrain configuration) - RW - Heave only profile created from real world encoder data (not run in drivetrain configuration) For those interested in the raw, unprocessed, data the authors have created a separate submission, linked below. This submission includes the raw TDMS files and associated files necessary to translate the data into either python or MATLAB formats. This data set has been developed by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided U.S. Department of Energy Office of Energy Efficiency and Renewable Energy Water Power Technologies Office.

16 TIDAL AND WAVE POWER↗
DOE OSTI2020

Big Data Processing for Power Grid Event Detection

In this paper we present the application of big data processing for the development of machine learning (ML) models to detect relevant events in power grid operations. This is based on almost 20TB of phasor measurement unit data corresponding to up to two years of operation of three grid interconnections which provide power to most of the United States. A significant aspect of the work consists in having all data processing performed on a single standard GPU server, from pre-processing to ML model training and testing. We describe the data and computational infrastructure, challenges faced and methods used in data processing, main findings and results. The ML approach employed for best utilization of the big data is also discussed, including sample results.

Paes Leao, Bruno↗