Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data virtualization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Behavior and mechanisms of Doppler wind lidar error in complex terrain: stable flow case study at Perdigão

A numerical experiment is carried out investigating the magnitude of biases in ground-based lidar measurements in complex flow conditions. Biases assessed include those arising from flow curvature and from the interaction of turbulence with the wind field reconstruction (WFR) algorithms used by a WindCube lidars and anemometers. RANS-CFD and WRF-LES simulations were performed for the Perdig˜ao Field Experiment site for a range of atmospheric conditions. Virtual anemometer and lidar data were generated for four locations: two near exposed ridge tops and two in low-speed regions in the valley. The LES data at these four locations show that the scalar inflation terms (the relation between scalar and vector averaged wind speed) for virtual lidar and virtual cups agree very well with predictions using perturbation theory. While the lidar errors vary greatly with location and height, the contribution from the flow curvature tends to be larger than the differences arising from scalar inflation. For one lidar/mast pair near the ridge top, comparisons between simulations and measurements are carried out for a resonant mountain wave event on June 14th, 2017, and for the whole duration of the Perdigão campaign for winds perpendicular to the ridges. The lidar error during the mountain wave, a period of strong stability and low inversion height, is significantly larger than the campaign average. The sensitivity of the lidar error to atmospheric stability is confirmed by the RANS simulations, which suggests strong sensitivity of flow curvature error to stability conditions and to the shape of the wind speed profile near the top of the boundary layer.

17 WIND ENERGY↗
NASA NTRS2019

Role of CEOS Working Group on Calibration and Validation in Analysis Ready Data Products

The Committee on Earth Observation Satellites (CEOS) is leading the CEOS Analysis Ready Data for Land (CARD4L) initiative. A goal of analysis ready data products is to limit the effort needed by users to pre-process the data allowing them to concentrate on the end products. CARD4L provides a set of specifications that data providers need to meet to be considered to satisfy CARD4L. One of the working groups within CEOS, the Working Group on Calibration and Validation (WGCV) is providing a peer review process to evaluate the documentation of the data providers validation and data product accuracy assessment. The approach makes use of the expertise within WGCV to collaborate with both the CEOS Land Surface Imaging Virtual Constellation and the data providers to work towards acceptance of the...

Thome, K.↗
NASA NTRS2020

State of the Art Small Spacecraft Technology

When the first edition of NASA’s Small Spacecraft Technology State-of-the-art report was published in 2013, 247 CubeSats and 105 other non-CubeSat small spacecraft under 50 kilograms (kg) had been launched, and these represented less than 2% of launched mass into orbit. By 2019, small spacecraft with mass less than 180 kg made up almost 7% of all mass launched into orbit. Additionally, 63% of spacecraft under 600 kg had mass less than 180 kg and of those 47% were CubeSats (1). Since 2013, flight heritage for small spacecraft, primarily CubeSats, has nearly doubled and with dedicated smallsat launch capabilities readily available and expanding, opportunities to demonstrate new technologies and systems are expected to increase. The 2020 edition of this report captures and distills the wealth of new information available on small spacecraft systems from NASA and other publicly available sources. Overall, this report is a survey of small spacecraft technologies sourced from open literature; it does not endeavor to be an original source, and only considers literature in the public domain to identify and classify devices. Commonly used sources for data include manufacturer datasheets, press releases, conference papers, journal papers, public filings with government agencies, news articles, presentations, and Small Spacecraft Systems Virtual Institute Federated Search. Data not appropriate for public dissemination, such as proprietary, export controlled, or otherwise restricted data, are not considered. As a result, this report includes many dedicated hours of desk research performed by subject matter experts reviewing resources noted above.

Bruce Yost↗
NASA NTRS2014

Climate Analytics as a Service

Climate science is a big data domain that is experiencing unprecedented growth. In our efforts to address the big data challenges of climate science, we are moving toward a notion of Climate Analytics-as-a-Service (CAaaS). CAaaS combines high-performance computing and data-proximal analytics with scalable data management, cloud computing virtualization, the notion of adaptive analytics, and a domain-harmonized API to improve the accessibility and usability of large collections of climate data. MERRA Analytic Services (MERRA/AS) provides an example of CAaaS. MERRA/AS enables MapReduce analytics over NASA's Modern-Era Retrospective Analysis for Research and Applications (MERRA) data collection. The MERRA reanalysis integrates observational data with numerical models to produce a global temporally and spatially consistent synthesis of key climate variables. The effectiveness of MERRA/AS has been demonstrated in several applications. In our experience, CAaaS is providing the agility required to meet our customers' increasing and changing data management and data analysis needs.

big data↗
DOE OSTI2021

Beyond optimization—supervised learning applications in relativistic laser-plasma experiments

We explore the applications of machine learning techniques in relativistic laser-plasma experiments beyond optimization purposes. We predict the beam charge of electrons produced in a laser wakefield accelerator given the laser wavefront change caused by a deformable mirror. Machine learning enables feature analysis beyond merely searching for an optimal beam charge, showing that specific aberrations in the laser wavefront are favored in generating higher beam charges. Supervised learning models allow characterizing the measured data quality as well as recognizing irreproducible data and potential outliers. Furthermore, we also include virtual measurement errors in the experimental data to examine the model robustness under these conditions. This work demonstrates how machine learning methods can benefit data analysis and physics interpretation in a highly nonlinear problem of relativistic laser-plasma interaction.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

ESS-DIVE Reporting Format for Sample-based Water and Soil Chemistry Measurements

The ESS-DIVE (Environmental Systems Science Data Infrastructure for a Virtual Ecosystem) reporting format for sample-based water and soil chemistry measurements are written guidelines and spreadsheet templates, that facilitate archiving water/soil/sediment chemistry data. The templates and terminology lists provided as part of the reporting format can help organize data and potentially enable data reuse.The reporting format consists of a general instructions file (instructions.md) as well as more detailed instructions files for each template (files beginning “Detailed_Instructions_*.md). The “examples” folder includes an example of each data template with just the limited set of required fields filled out as well as other examples with both required and optional fields complete.The 'templates' folder contains CSV templates for data, methods, and terminology file templates. Similar to the examples folder, the blank templates are provided with versions for required fields only as well as required and optional fields. Lastly, the term list folder provides common terminology used in each of the templates as well as a definition and any constraints for the fields.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

ESS-DIVE Reporting Format for Dataset Package Metadata

ESS-DIVE’s (Environmental Systems Science Data Infrastructure for a Virtual Ecosystem) dataset metadata reporting format is intended to compile information about a dataset (e.g., title, description, funding sources) that can enable reuse of data submitted to the ESS-DIVE data repository. The files contained in this dataset include instructions (dataset_metadata_guide.md and README.md) that can be used to understand the types of metadata ESS-DIVE collects. The data dictionary (dd.csv) follows ESS-DIVE’s file-level metadata reporting format and includes brief descriptions about each element of the dataset metadata reporting format. This dataset also includes a terminology crosswalk (dataset_metadata_crosswalk.csv) that shows how ESS-DIVE’s metadata reporting format maps onto other existing metadata standards and reporting formats.Data contributors to ESS-DIVE can provide this metadata by manual entry using a web form or programmatically via ESS-DIVE’s API (Application Programming Interface). A metadata template (dataset_metadata_template.docx or dataset_metadata_template.pdf) can be used to collaboratively compile metadata before providing it to ESS-DIVE.Since being incorporated into ESS-DIVE’s data submission user interface, ESS-DIVE’s dataset metadata reporting format, has enabled features like automated metadata quality checks, and dissemination of ESS-DIVE datasets onto other data platforms including Google Dataset Search and DataCite.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1983

Automatic-repeat-request error control schemes

Error detection incorporated with automatic-repeat-request (ARQ) is widely used for error control in data communication systems. This method of error control is simple and provides high system reliability. If a properly chosen code is used for error detection, virtually error-free data transmission can be attained. Various types of ARQ and hybrid ARQ schemes, and error detection using linear block codes are surveyed.

Lin, S.↗
NASA NTRS1989

Prototype architecture for a VLSI level zero processing system

The prototype architecture and implementation of a high-speed level zero processing (LZP) system are discussed. Due to the new processing algorithm and VLSI technology, the prototype LZP system features compact size, low cost, high processing throughput, and easy maintainability and increased reliability. Though extensive control functions have been done by hardware, the programmability of processing tasks makes it possible to adapt the system to different data formats and processing requirements. It is noted that the LZP system can handle up to 8 virtual channels and 24 sources with combined data volume of 15 Gbytes per orbit. For greater demands, multiple LZP systems can be configured in parallel, each called a processing channel and assigned a subset of virtual channels. The telemetry data stream will be steered into different processing channels in accordance with their virtual channel IDs. This super system can cope with a virtually unlimited number of virtual channels and sources. In the near future, it is expected that new disk farms with data rate exceeding 150 Mbps will be available from commercial vendors due to the advance in disk drive technology.

Shi, Jianfei↗
NASA NTRS2015

Architectures Toward Reusable Science Data Systems

Science Data Systems (SDS) comprise an important class of data processing systems that support product generation from remote sensors and in-situ observations. These systems enable research into new science data products, replication of experiments and verification of results. NASA has been building systems for satellite data processing since the first Earth observing satellites launched and is continuing development of systems to support NASA science research and NOAAs Earth observing satellite operations. The basic data processing workflows and scenarios continue to be valid for remote sensor observations research as well as for the complex multi-instrument operational satellite data systems being built today. System functions such as ingest, product generation and distribution need to be configured and performed in a consistent and repeatable way with an emphasis on scalability. This paper will examine the key architectural elements of several NASA satellite data processing systems currently in operation and under development that make them suitable for scaling and reuse. Examples of architectural elements that have become attractive include virtual machine environments, standard data product formats, metadata content and file naming, workflow and job management frameworks, data acquisition, search, and distribution protocols. By highlighting key elements and implementation experience we expect to find architectures that will outlast their original application and be readily adaptable for new applications. Concepts and principles are explored that lead to sound guidance for SDS developers and strategists.

Data Processing↗
DOE OSTI2025

SCILLA Scanning Electrical Mobility Spectrometer (SEMS) airborne data

The Scanning Electrical Mobility Spectrometer (SEMS) provides measurements of bulk aerosol size distributions, which are crucial for studying aerosol-cloud interactions. The instrument sampled through two different inlets: an isokinetic inlet and a counter-flow virtual impactor (CVI) inlet. Data were collected to help improve climate and earth systems models, characterize atmospheric conditions, and validate satellite observations and climate simulations.

Scanning Electrical Mobility Spectrometer↗
DOE OSTI2025

Machine Learning-Assisted Recovery of Delicate Kinetic Information from Transient Reactor Experiments

Identifying active sites and their roles in chemical reaction steps remains a vital challenge in heterogeneous catalysis. Transient experiments offer a unique way to probe active sites and distinguish subtle kinetic features. Although physics-based analysis methods may be well-developed, they can be highly susceptible to experimental noise, and smoothing methods may erase or even distort important features; a smooth curve is not always the best curve. We demonstrate a new workflow for the direct interpretation of intrinsic kinetic information from exit flux curves measured in transient reactor experiments. This workflow contains three artificial neural networks (ANNs), including a noise reducer, a concentration predictor, and a rate predictor to analyze experimental data, followed by the virtual TAP (VTAP) physics-based reactor model and density functional theory (DFT) calculations of adsorption energies on specific sites. We use this workflow to analyze the data from experiments titrating Pt/Al 2 O 3 and Pt/SiO 2 catalysts with carbon monoxide (CO) in the temporal analysis of products (TAP) reactor. Our workflow separates the time-evolving chemical reaction and mass transfer information contained in the TAP pulse response. The existence of strong- and weak-binding sites on the Pt/Al 2 O 3 catalyst is observed in the catalyst titration experiment in the transient reactor. The structures of the strong- and weak-binding sites are then identified by using DFT calculations. We find that the Pt/SiO 2 catalyst has only strong-binding sites, which aligns with the inactive support effect of SiO 2 . We demonstrate how machine learning methods provide unique insights with high-resolution data analysis that cannot be achieved by using state-of-the-art physics-based methods.

Adsorption↗
NASA NTRS2023

First Principles Modeling of the Thermal Amine Scrubber Flight Experiment’s Chemical Performance

The removal of atmospheric CO2 from a spacecraft is of particular importance to NASA’s mission, and is an area of continual study and technological advancement. One of the more recent advancements has been with reusable sorbents being regenerated with a combination of heat and vacuum. One such technology is the Thermal Amine Scrubber (TAS) flight experiment on the ISS, though several others are currently flying or preparing to fly. A model was created of the TAS to predict chemical performance, using fundamental chemistry and physics based on principles rather than empirical relations. Since the physical laws are true across all conditions, such a model enables greater model accuracy outside the bounds of test data, and allows for virtual testing of the hardware at conditions that are prohibitively difficult or expensive to actually test. This paper details the model’s development, operation, and correlation to data from the flight unit. The model is then compared to a data set taken from the flight unit under different flow, CO2 partial pressure, and bed configuration conditions, resulting in only a 2% error. The equations and principles laid forth in this paper are applicable to a wide range of thermally regenerated sorbents, and additional models of a similar nature would allow for potentially the most straightforward and direct method of comparison of technologies available to date.

CO2 removal↗
DOE OSTI2020

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Scientific applications at exascale generate and analyze massive amounts of data. A critical requirement of these applications is the capability to access and manage this data efficiently on exascale systems. As such, parallel I/O, the key technology enables moving data between compute nodes and storage, faces monumental challenges from new applications, memory, and storage architectures considered in the designs of exascale systems. As the storage hierarchy is expanding to include node-local persistent memory, burst buffers, etc., as well as disk-based storage, data movement among these layers must be efficient. Parallel I/O libraries of the future should be capable of handling file sizes of many terabytes and beyond. In this paper, we describe new capabilities we have developed in Hierarchical Data Format version 5 (HDF5), the most popular parallel I/O library for scientific applications. HDF5 is one of the most used libraries at the leadership computing facilities for performing parallel I/O on existing HPC systems. The state-of-the-art features we describe include: Virtual Object Layer (VOL), Data Elevator, asynchronous I/O, full-featured single-writer and multiple-reader (Full SWMR), and parallel querying. In this paper, we introduce these features, their implementations, and the performance and feature benefits to applications and other libraries.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Building Capacity for Policy-makers in a Virtual Setting: Providing Tools to Analyze Wildfire Smoke Plumes and Their Impacts

The NASA DEVELOP Program conducted 10-week long feasibility projects in a remote work setting, including partnering with The Nature Conservancy’s Washington Chapter and the Puget Sound Clean Air Agency to investigate wildfire smoke from 2000 - 2020 in the Pacific Northwest using satellite-derived data. The team engaged with platforms for collaboration both internally with NASA affiliates and externally with community organizations. Working from multiple states, the team members used a variety of software including Google Meet, Microsoft Teams, and Google Earth Engine to foster communication and work with data in a shared virtual environment. Throughout the project, the team learned that executing the project in a distanced work setting made it easier to reach out to scientists across the country for expertise and guidance. To study changes in air quality resulting from wildfire smoke, the team utilized data from NASA’s Fire Information from Resource Management System (FIRMS) and the ESA’s Sentinel-5 TROPOspheric Monitoring Instrument (TROPOMI). The team created a Google Earth Engine web-based tool, “Plume Hazards and Observations of Emissions by Navigating an Interactive eXplorer” (PHOENIX), to visualize changes in pollutants and aerosol optical depth after fire events. The potential relationship between plume height and fire radiative power was evaluated by using NASA’s Moderate Resolution Imaging Spectroradiometer (MODIS) aboard Aqua and Terra satellites and NASA’s Multi-angle Imaging SpectroRadiometer (MISR) aboard Terra with the MISR INteractive eXplorer (MINX). The PHOENIX smoke assessment tool and science communication infographics will be shared electronically with the partner organizations. Furthermore, the team introduced the partners to MINX and will provide a tailored tutorial that included a recorded video and a written component with a live virtual workshop. These resources build capacity for further research and education on wildfire smoke and air quality within communities.

NASA DEVELOP↗
DOE OSTI2025

Label-based Virtual Directories In dCache

Traditional filesystems organize data in directories. These directories are typically a collection of files whose grouping is based on a single criterion, e.g., the starting date of an experiment, experiment name, beamline ID, measurement device, or instrument. However, each file in a directory can belong to several logical groups, such as a special event type, experiment condition, or a part of a selected dataset. dCache is a storage system developed to store large amounts of scientific data, used by many HEP and Photon Science experiments. With recent developments in dCache, we have introduced a concept of file tagging, which dynamically groups files with the same label into virtual directories. The file labels can be added, removed, renamed, and deleted through the admin interface or via REST API. The files in virtual directories are exposed through all protocols supported by dCache. This contribution will describe the details of the implementation for file tagging in dCache and present our future development plans on automatic metadata extractions, a feature that will significantly simplify data management. Additionally, we are exploring the future use of virtual directories as a way to translate scientific data catalogs into filesystem views for direct data analysis.

Sahakyan, Marina [DESY]↗
DOE OSTI2024

Geospatial Data Platform for All

Spatiotemporal data has evolved in scale due to augmented use in cross-domain applications. Simultaneously, there is substantial growth in the availability of Geographic Information Systems (GIS) data provided by the United States Geological Survey (USGS) along with other federal, state, county, or local agencies through open-data portals and public access APIs. However, data availability does not equate with accessibility. Large-scale analyses and applications require robust, performant data management with co-location of data storage and computing. The insufficiency of data management infrastructure compels researchers to adopt ad hoc project- specific GIS data storage solutions (e.g., copying data to High-Performance computer file systems). As an ad hoc storage strategy does not scale, it hampers cross-domain analyses causing difficulty in data reuse and utilizing existing code bases. Furthermore, GIS data is complex and requires expertise to analyze and manipulate due to its intricate data structures and data-specific projection transformations. Despite the challenges, we recognize that derived GIS data products, e.g., satellite or LIDAR-based images, can be used in downstream applications such as AI by domain, but non-GIS experts. To address the data needs and overcome the challenges, we are working towards a GIS Data Platform focused on efficient data storage, data discovery and access, and an API to enable common workflows. We propose a knowledge-graph (KG) approach for data discovery, whereby datasets are semantically linked to higher- level constructs such as projects and research areas. The semantic data links enable researchers to explore datasets in a top-down approach by specifying relevant and meaningful terms (assists in finding hidden data). An advantage is that the nodes and edges in a knowledge graph create built-in semantic documentation. Deeper spatiotemporal connections between data sources can be encoded via Graph Neural Networks (GNN) (Zhang et al., 2021). The KG approach can be extended to integrate the data itself in a Virtual KG (VKG). Our work will derive inspiration from large-scale VKG efforts that have been undertaken or are currently underway as part of the OpenStreetMap project (Ding et al., 2021). For DOE Data Days, we share the proposed geospatial data platform hybrid (cloud/on-prem) architecture, our work-to-date on storing, retrieving, and transforming LiDAR and raster data relevant to two important NREL use-cases, including the Renewable Energy Potential (reV) Model, and present our proposal for a KG based data discovery engine.

data platform↗