Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data processing and standardization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Enabling modern data discovery for atmospheric measurements

The Atmospheric Radiation Measurement (ARM) user facility is a US Department of Energy Office of Science user facility that is managed and operated through a collaborative effort led by nine US Department of Energy national laboratories. The ARM Data Center, located at Oak Ridge National Laboratory, is responsible for the timely collection, processing, and delivery of data products to the scientific community. The ARM Data Center holds more than 11,000 data products, including metadata collected from field campaigns, instruments, value-added products, and principal investigator–contributed data. These data sets are checked for successful transfer (for most data, this transfer is carried out automatically via the network; however, some of the largest data sets and some of the most remote sites require manual shipping of hard disks) and both the data and metadata are processed to a standard format, which is an ARM-standardized structure, via the Network Common Data Form. The Network Common Data Form is a self-describing binary format with many compatible software tools. Once processed, the data are cataloged, stored in the ARM Data Archive, and made discoverable through association with an array of metadata-characterizing information, such as location and measurement classification. These metadata enable powerful search capabilities through the ARM Data Center Data Discovery interface. This paper discusses the workflow of how the new discovery system has been redesigned from user requirements and how the data are distributed to the scientific community.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1995

Visualization tools for the processing of airglow data from RAIDS

In anticipation of large data sets associated with a number of atmospheric imaging instruments being prepared for long term global coverage, NRL is developing graphical interfaces for all aspects of the program. For the first of these projects, RAIDS (the Remote Atmospheric and Ionospheric Detection System), a graphical approach to data handling, visualization, and analysis is envisioned and will set the stage for the satellites that follow. An overall system of hardware and a set of software 'tools,' that will allow for both the routine handling of all data and the analysis of large data sets assembled by scientists and instrument engineers, are currently being developed. The software for standard processing and visualization of instrument data is independent of computer platform and will allow for easy adaptation from one experiment to another. The processing will produce data sets that have similar characteristics, allowing for easy comparison of data obtained under similar circumstances. The visualization of both the engineering and scientific data is an important part of the system. By creating graphical environments for engineering evaluations and for scientific analysis data sets can be viewed and analyzed rapidly. This rapid analysis of data will contribute towards a greater portion of the RAIDS data being utilized.

Miller, Gordon J.↗
NASA NTRS1992

A sophisticated, multi-channel data acquisition and processing system for high frequency noise research

A sophisticated, multi-channel computerized data acquisition and processing system was developed at the NASA LeRC for use in noise experiments. This technology, which is available for transfer to industry, provides a convenient, cost-effective alternative to analog tape recording for high frequency acoustic measurements. This system provides 32-channel acquisition of microphone signals with an analysis bandwidth up to 100 kHz per channel. Cost was minimized through the use of off-the-shelf components. Requirements to allow for future expansion were met by choosing equipment which adheres to established industry standards for hardware and software. Data processing capabilities include narrow band and 1/3 octave spectral analysis, compensation for microphone frequency response/directivity, and correction of acoustic data to standard day conditions. The system was used successfully in a major wind tunnel test program at NASA LeRC to acquire and analyze jet noise data in support of the High Speed Civil Transport (HSCT) program.

Hall, David G.↗
NASA NTRS2010

The Global Precipitation Measurement (GPM) Project

The Global Precipitation Measurement (GP!v1) mission is an international cooperative effort to advance the understanding of the physics of the Earth's water and energy cycle. Accurate and timely knowledge of global precipitation is essential for understanding the weather/climate/ecological system, for improving our ability to manage freshwater resources, and for predicting high-impact natural hazard events including floods, droughts, extreme weather events, and landslides. The GPM Core Observatory will be a reference standard to uniformly calibrate data from a constellation of spacecraft with passive microwave sensors. GPM is being developed under a partnership between the United States (US) National Aeronautics and Space Administration (NASA) and the Japanese Aerospace and Exploration Agency (JAXA). NASA's Goddard Space Flight Center (GSFC), in Greenbelt, MD is developing the Core Observatory, two GPM Microwave Imager (GMI) instruments, Ground Validation System and Precipitation Processing System for the GPM mission. JAXA will provide a Dual-frequency Precipitation Radar (DPR) for installation on the Core satellite and launch services for the Core Observatory. The second GMI instrument will be flown on a partner-provided spacecraft. Other US agencies and international partners contribute to the GPM mission by providing precipitation measurements obtained from their own spacecraft and/or providing ground-based precipitation measurements to support ground validation activities. The Precipitation Processing System will provide standard data products for the mission.

Azarbarzin, Ardeshir↗
DOE OSTI2021

Accuracy of Vaisala RS41 and RS92 Upper Tropospheric Humidity Compared to Satellite Hyperspectral Infrared Measurements

Radiosondes are important for calibrating satellite sensors and assessing sounding retrievals. Vaisala RS41 radiosondes have mostly replaced RS92 in the Global Climate Observing System (GCOS) Reference Upper Air Network (GRUAN) and the conventional network. This study assesses RS41 and RS92 upper tropospheric humidity (UTH) accuracy by comparing with Infrared Atmospheric Sounding Interferometer (IASI) upper tropospheric water vapor absorption spectrum measurements. Using single RS41 and RS92 soundings at three GRUAN and DOE Atmospheric Radiation Measurement (ARM) sites and dual RS92/RS41 launches at three additional GRUAN sites, collocated with cloud-free IASI radiances (OBS), we compute Line-by-Line Radiative Transfer Model radiances for radiosonde profiles (CAL). We analyze OBS-CAL differences from 2015 to 2020, for daytime, nighttime, and dusk/dawn separately if data is available, for standard (STD) RS92 and RS41 processing, and RS92 GRUAN Data Processing (GDP; RS41 GDP is in development). We find that daytime RS41 (even without GDP) has ~1% smaller UTH errors than GDP RS92. RS41 may still have a dry bias of 1–1.5% for both daytime and nighttime, and a similar error for nighttime RS92 GDP, while standard RS92 may have a dry bias of 3–4%. These sonde humidity biases are probably upper limits since “cloud-free” scenes could still be cloud contaminated. Radiances computed from European Centre for Medium-Range Weather Forecasts (ECMWF) analyses match better than radiosondes with IASI measurements, perhaps because ECMWF assimilates IASI measurements. Relative differences between RS41 STD and RS92 GDP, or between radiosondes and ECMWF humidity profiles obtained from the radiance analysis, are consistent with their differences obtained directly from the RH measurements.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Big Data Processing for Power Grid Event Detection

In this paper we present the application of big data processing for the development of machine learning(ML) models to detect relevant events in power gridoperations. This is based on almost 20TB of phasormeasurement unit data corresponding to up to two years of operation of three grid interconnections which provide power to most of the United States. A significant aspect of the work consists in having all data processing performed on a single standard GPU server, from pre-processing to ML model training and testing. We describe the data and computational infrastructure, challenges faced and methods used in dataprocessing, main findings and results. The ML approach employed for best utilization of the big data is also discussed, including sample results.

Paes Leao, Bruno↗
DOE OSTI2020

Big Data Processing for Power Grid Event Detection

In this paper we present the application of big data processing for the development of machine learning (ML) models to detect relevant events in power grid operations. This is based on almost 20TB of phasor measurement unit data corresponding to up to two years of operation of three grid interconnections which provide power to most of the United States. A significant aspect of the work consists in having all data processing performed on a single standard GPU server, from pre-processing to ML model training and testing. We describe the data and computational infrastructure, challenges faced and methods used in data processing, main findings and results. The ML approach employed for best utilization of the big data is also discussed, including sample results.

Paes Leao, Bruno↗
DOE OSTI2025

Ring Pull Strain Analysis Version 1.1

This report details an analysis package, Ring Pull Strain Analysis (RPSA), that can be used to present and quantify digital image correlation (DIC) data as it relates to a gaugeless ring pull test. Gaugeless ring pull is a testing technique for mechanical testing of small annular samples, usually cut from a thin-walled tube. DIC data is often necessary for this kind of test because bending moments present on the ring cause a non-uniform strain distribution and localized measurements are necessary. In addition, the annular geometry of a ring lends itself to a polar representation, which is not present with typical DIC analysis methods. RPSA was made to calculate and plot the polar representation of strain from standard pre-processed DIC data of a gaugeless ring pull test. Further analysis can be done on ring pull including a quasi-uniaxial tensile analysis and coating analysis, which are also performed by RPSA. In addition, due to the universality of DIC plotting and ring pull test analysis, RPSA can accommodate a wide variety of tests, though it is tailored for ring pull testing. This report details how RPSA works, including the theory, assumptions, and logic behind the calculations and the structure of the program.

36 MATERIALS SCIENCE↗
NASA NTRS2011

Experiment in Onboard Synthetic Aperture Radar Data Processing

Single event upsets (SEUs) are a threat to any computing system running on hardware that has not been physically radiation hardened. In addition to mandating the use of performance-limited, hardened heritage equipment, prior techniques for dealing with the SEU problem often involved hardware-based error detection and correction (EDAC). With limited computing resources, software- based EDAC, or any more elaborate recovery methods, were often not feasible. Synthetic aperture radars (SARs), when operated in the space environment, are interesting due to their relevance to NASAs objectives, but problematic in the sense of producing prodigious amounts of raw data. Prior implementations of the SAR data processing algorithm have been too slow, too computationally intensive, and require too much application memory for onboard execution to be a realistic option when using the type of heritage processing technology described above. This standard C-language implementation of SAR data processing is distributed over many cores of a Tilera Multicore Processor, and employs novel Radiation Hardening by Software (RHBS) techniques designed to protect the component processes (one per core) and their shared application memory from the sort of SEUs expected in the space environment. The source code includes calls to Tilera APIs, and a specialized Tilera compiler is required to produce a Tilera executable. The compiled application reads input data describing the position and orientation of a radar platform, as well as its radar-burst data, over time and writes out processed data in a form that is useful for analysis of the radar observations.

Holland, Matthew↗
NASA NTRS2020

Application of IRIG 106 Digital Data Recorder Standards for Flight Test

During flight tests, data is acquired from different sources and in different formats. Sensor data, aircraft avionics bus data, time, voice and video are essential for analysis of these flight tests. IRIG 106 standards exist for recording and transferring these data types. However, to enhance interoperability of flight test data between (multinational) test teams, a standard was needed to encapsulate all these data types. For this purpose the Telemetry Group of the Range Commanders Council (RCC) first published in 2004 the IRIG 106 Chapter 10 Standard, “Solid-State On-Board Recorder Standard”. This standard is evolving up to present. This AGARDograph gives an introduction to the standard, explains how the standard is positioned among other relevant standards, and describes the equipment and their application in flight tests with some examples. The topics covered include IRIG 106 Chapter 10 compatible recording of (multiple) data streams, synchronization of data, data recorders, data retrieval, data processing, and examples of flight test programs. Further development of the standard, originally published for on-board recorders, is described for extending the requirements for ground-based data recorders. The last chapter addresses a number of issues related to the application of the Chapter 10 standard with references to information and possible solutions. This document shows how data acquisition and data processing in accordance to IRIG 106 Chapter 10 standard can be applied in NATO flight tests and will thereby contribute to collaboration and cost efficiency.

flight test instrumentation↗
NASA NTRS1991

Advanced techniques and technology for efficient data storage, access, and transfer

Advanced techniques for efficiently representing most forms of data are being implemented in practical hardware and software form through the joint efforts of three NASA centers. These techniques adapt to local statistical variations to continually provide near optimum code efficiency when representing data without error. Demonstrated in several earlier space applications, these techniques are the basis of initial NASA data compression standards specifications. Since the techniques clearly apply to most NASA science data, NASA invested in the development of both hardware and software implementations for general use. This investment includes high-speed single-chip very large scale integration (VLSI) coding and decoding modules as well as machine-transferrable software routines. The hardware chips were tested in the laboratory at data rates as high as 700 Mbits/s. A coding module's definition includes a predictive preprocessing stage and a powerful adaptive coding stage. The function of the preprocessor is to optimally process incoming data into a standard form data source that the second stage can handle.The built-in preprocessor of the VLSI coder chips is ideal for high-speed sampled data applications such as imaging and high-quality audio, but additionally, the second stage adaptive coder can be used separately with any source that can be externally preprocessed into the 'standard form'. This generic functionality assures that the applicability of these techniques and their recent high-speed implementations should be equally broad outside of NASA.

Rice, Robert F.↗
DOE OSTI2025

Fostering Geothermal Machine Learning Success: Elevating Big Data Accessibility and Automated Data Standardization in the Geothermal Data Repository

The Department of Energy's (DOE's) Geothermal Data Repository (GDR) has implemented improvements to both its data lakes and its data standards and automated data pipelines. The GDR data lakes have reduced storage and compute-related barriers to using large geothermal datasets, enabling these large datasets to be accessed by anyone with a modern computer and internet access. More recently, the GDR has been working to further reduce barriers through streamlining the data intake process, educating users on the process and requirements, and helping users access data from the data lakes. These improvements have augmented the quantity of datasets the GDR is able to accept into its data lakes and have enabled users who are new to cloud tools to access these datasets more easily, overall increasing the accessibility of big geothermal data for use in machine learning and other projects. In addition, the GDR now has built-in data standards and pipelines for drilling data, geospatial data, and distributed acoustic sensing (DAS) data. These standardization efforts aim to enhance the real-world applicability of geothermal machine learning outcomes by improving the quality of training data. Specifically, through standardizing high-value datasets, the GDR is reducing project-specific data curation requirements, thus allowing more time for actual research. By automating this process, the burden of standardization is lifted from the user, ultimately increasing the availability of standardized data. This paper provides an update on recent improvements made to the GDR's data lakes and automated data pipelines, including: (1) streamlining the data lake intake process, (2) better educating users on the process and requirements through a new data lakes page, (3) adding data lake direct access links to GDR data lake submission pages, (4) implementing a DAS data pipeline to convert DAS data uploaded in SEG-Y format to a standardized hierarchical data format v5 (HDF5), (5) extending this pipeline to encompass data in the GDR data lake, (6) adding metadata requirements for geospatial data, (7) making user interface/user experience (UX) enhancements to the data pipelines' documentation pages, and (8) improving the GDR's data standards and pipelines pages to better guide users in ensuring that their data is standardized by the GDR's automated data pipelines. 2024 Geothermal Resources Council. All rights reserved.

accessibility↗
DOE OSTI2023

AmeriFlux BASE data pipeline to support network growth and data sharing

Abstract AmeriFlux is a network of research sites that measure carbon, water, and energy fluxes between ecosystems and the atmosphere using the eddy covariance technique to study a variety of Earth science questions. AmeriFlux’s diversity of ecosystems, instruments, and data-processing routines create challenges for data standardization, quality assurance, and sharing across the network. To address these challenges, the AmeriFlux Management Project (AMP) designed and implemented the BASE data-processing pipeline. The pipeline begins with data uploaded by the site teams, followed by the AMP team’s quality assurance and quality control (QA/QC), ingestion of site metadata, and publication of the BASE data product. The semi-automated pipeline enables us to keep pace with the rapid growth of the network. As of 2022, the AmeriFlux BASE data product contains 3,130 site years of data from 444 sites, with standardized units and variable names of more than 60 common variables, representing the largest long-term data repository for flux-met data in the world. The standardized, quality-ensured data product facilitates multisite comparisons, model evaluations, and data syntheses.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS2014

The Parallel System for Integrating Impact Models and Sectors (pSIMS)

We present a framework for massively parallel climate impact simulations: the parallel System for Integrating Impact Models and Sectors (pSIMS). This framework comprises a) tools for ingesting and converting large amounts of data to a versatile datatype based on a common geospatial grid; b) tools for translating this datatype into custom formats for site-based models; c) a scalable parallel framework for performing large ensemble simulations, using any one of a number of different impacts models, on clusters, supercomputers, distributed grids, or clouds; d) tools and data standards for reformatting outputs to common datatypes for analysis and visualization; and e) methodologies for aggregating these datatypes to arbitrary spatial scales such as administrative and environmental demarcations. By automating many time-consuming and error-prone aspects of large-scale climate impacts studies, pSIMS accelerates computational research, encourages model intercomparison, and enhances reproducibility of simulation results. We present the pSIMS design and use example assessments to demonstrate its multi-model, multi-scale, and multi-sector versatility.

crop modeling↗
DOE OSTI2024

Identifying Sample Provenance From SEM/EDS Automated Particle Analysis via Few-Shot Learning Coupled With Similarity Graph Clustering

Automated particle analysis (APA) provides a vast amount of compositional data via energy-dispersive X-ray spectroscopy along with size and shape data via scanning electron microscopy for individual particles in a sample. In many instances, APA data are leveraged to support identification of the source of a sample based on the detection of particles of a specific composition. Often, the particles that provide context make up a minuscule portion of the sample. Additionally, the interpretation of complex samples can be difficult due to the diversity of compositions both in the mixture and within a particle. In this work, we demonstrate a method to compute and cluster similarity graphs that describe inter-particle relationships within a sample using a multi-modal few-shot learning neural network. Here, as a proof-of-concept, we show that samples known to have been exposed to gunshot residue can be distinguished from samples occasionally mistaken for gunshot residue. Our workflow builds upon standard APA techniques and data processing methods to unveil additional information in a readily interpretable and quantitatively comparable format.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

CalWave - Open Water Demonstration - Field Testing Content Models (10/21 - 7/22)

Data for the CalWave - Open Water Demonstration, a submerged pressure differential Wave Energy Converter (WEC) Device. Device is moored to the seabed, and the motion of the waves causes the sea level to rise and fall above the device, inducing a pressure differential in the device. The alternating pressure pumps fluid through a system to generate electricity, which is transmitted to shore via bidirectional cables. Documentation and data here includes: Open Water Demonstration, including field testing content models for nearly 10 months of continuous ocean operation, from October 2021 through July 2022.

16 TIDAL AND WAVE POWER↗
DOE OSTI2026

Inter-year Variability in EDS Standards

This report provides a deep dive into how the stability of energy dispersive spectroscopy (EDS) standards yielded insight into the shortfalls of using software to randomly select points for spectral acquisition when using heterogeneous standards. Some procedural recommendations are included which are meant to minimize the impact of anomalous reference standard spectra by detecting them before the standard is implemented into data processing.

36 MATERIALS SCIENCE↗
NASA NTRS1979

Application of analytical redundancy management to Shuttle crafts

The study involves the bank of filters approach to analytical redundancy management since this is amenable to microelectronic implementation. Attention is given to a study of the UD factorized filter to determine if it gives more accurate estimates than the standard Kalman filter when data processing word size is reduced. It is reported that, as the word size is reduced, the effect of modeling error dominates the filter performance of the two filters. However, the UD filter is shown to maintain a slight advantage in tracking performance. It is concluded that because of the UD filter's stability in the serial processing mode, it remains the leading candidate for microelectronic implementation.

Montgomery, R. C.↗