Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “hierarchical data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Database Design Strategies for Coordinated Simulation and Testing in Additive Manufacturing

The qualification and certification (Q&C) process presents a significant challenge for widespread adoption of additive manufacturing (AM) materials and processes for aerospace applications. A relational database framework will be presented as a tool for data curation of coordinated experimental and computational materials modeling research activities. A comparison of relational and hierarchical data structures in this domain will be emphasized through the evolution of a database design strategy. This framework’s mission is to support the advancement of computational materials-informed Q&C by providing the necessary data infrastructure to trace reliability and reproducibility measures through unified AM materials simulation and experimental testing. FAIR (findable, accessible, interoperable, and reusable) data will be highlighted as a necessary precursor for automation of specific actions, which ultimately reduces the time and expense burden for Q&C. The discussion will be mostly limited to back-end design elements, though a few front-end user experience examples will also be shared.

Qualification↗
NASA NTRS1997

The Cloud Absorption Radiometer HDF Data User's Guide

The purpose of this document is to describe the Cloud Absorption Radiometer (CAR) Instrument, methods used in the CAR Hierarchical Data Format (HDF) data processing, the structure and format of the CAR HDF data files, and methods for accessing the data. Examples of CAR applications and their results are also presented. The CAR instrument is a multiwavelength scanning radiometer that measures the angular distributions of scattered radiation.

Li, Jason Y.↗
NASA NTRS1998

Locating Biodiversity Data Through The Global Change Master Directory

The Global Change Master Directory (GCMD) presently holds descriptions for almost 7000 data sets held worldwide. The directory's primary purpose is for data discovery. The information provided through the GCMD's Directory Interchange Format (DIF) is the set of information that a researcher would need to determine if a particular data set could be of value. By offering data set descriptions worldwide in many scientific disciplines - including meteorology, oceanography, ecology, geology, hydrology, geophysics, remote sensing, paleoclimate, solar-terrestrial physics, and human dimensions of climate change - the GCMD simplifies the discovery of data sources. Direct linkages to many of the data sets are also provided. In addition, several data set registration tools are offered for populating the directory. To search the directory, one may choose the Guided Search or Free-Text Search. Two experimental interfaces were also made available with the latest software release - one based on a keyword search and another based on a graphical interface. The graphical interface was designed in collaboration with the Human Computer Interaction Laboratory at the University of Maryland. The latest version of the software, Version 6, was released in April, 1998. It features the implementation of a scheme to handle hierarchical data set collections (parent-child relationships); a hierarchical geospatial location search scheme; a Java-based geographic map for conducting geospatial searches; a Related-URL field for project-related data set collections, metadata extensions (such as more detailed inventory information), etc.; a new implementation of the Isite software; a new dataset language field; hyperlinked email addresses, and more. The key to the continued evolution of the GCMD is in the flexibility of the GCMD database, allowing modifications and additions to made relatively easily to maintain currency, thus providing the ability to capitalize on current technology while importing all existing records. Changes are discussed and approved through an online "interoperability" forum. The next major release of the GCMD is scheduled for early 1999 and will include the incorporation of a new matrix-based interface, a rapid valids-based query system; improvement in the operations facility - important for future distributed options; new streamlined code for greater performance and maintainability; improvements in the handling of seven current fields proposed through the interoperability forum (at no expense to the data providers); and the release of DOCmorph, a more robust version of DIFmorph to translate many 'standards' multi-directionally. Issues and actions will also be addressed.

Olsen, Lola M.↗
NASA NTRS1994

SeaWiFS technical report series. Volume 19: Case studies for SeaWiFS calibration and validation, part 2

This document provides brief reports, or case studies, on a number of investigations and data set development activities sponsored by the Calibration and Validation Team (CVT) within the Sea-viewing Wide Field-of-view Sensor (SeaWiFS) Project. Chapter 1 is a comparison with the atmospheric correction of Coastal Zone Color Scanner (CZCS) data using two independent radiative transfer formulations. Chapter 2 is a study on lunar reflectance at the SeaWiFS wavelengths which was useful in establishing the SeaWiFS lunar gain. Chapter 3 reports the results of the first ground-based solar calibration of the SeaWiFS instrument. The experiment was repeated in the fall of 1993 after the instrument was modified to reduce stray light; the results from the second experiment will be provided in the next case studies volume. Chapter 4 is a laboratory experiment using trap detectors which may be useful tools in the calibration round-robin program. Chapter 5 is the original data format evaluation study conducted in 1992 which outlines the technical criteria used in considering three candidate formats, the hierarchical data format (HDF), the common data format (CDF), and the network CDF (netCDF). Chapter 6 summarizes the meteorological data sets accumulated during the first three years of CZCS operation which are being used for initial testing of the operational SeaWiFS algorithms and systems and would be used during a second global processing of the CZCS data set. Chapter 7 describes how near-real time surface meteorological and total ozone data required for the atmospheric correction algorithm will be retrieved and processed. Finally, Chapter 8 is a comparison of surface wind products from various operational meteorological centers and field observations. Surface winds are used in the atmospheric correction scheme to estimate glint and foam radiances.

Hooker, Stanford B.↗
NASA NTRS2006

Improvement in Recursive Hierarchical Segmentation of Data

A further modification has been made in the algorithm and implementing software reported in Modified Recursive Hierarchical Segmentation of Data (GSC- 14681-1), NASA Tech Briefs, Vol. 30, No. 6 (June 2006), page 51. That software performs recursive hierarchical segmentation of data having spatial characteristics (e.g., spectral-image data). The output of a prior version of the software contained artifacts, including spurious segmentation-image regions bounded by processing-window edges. The modification for suppressing the artifacts, mentioned in the cited article, was addition of a subroutine that analyzes data in the vicinities of seams to find pairs of regions that tend to lie adjacent to each other on opposite sides of the seams. Within each such pair, pixels in one region that are more similar to pixels in the other region are reassigned to the other region. The present modification provides for a parameter ranging from 0 to 1 for controlling the relative priority of merges between spatially adjacent and spatially non-adjacent regions. At 1, spatially-adjacent-/spatially- non-adjacent-region merges have equal priority. At 0, only spatially-adjacent-region merges (no spectral clustering) are allowed. Between 0 and 1, spatially-adjacent- region merges have priority over spatially- non-adjacent ones.

Tilton, James C.↗
DOE OSTI2020

A Python Module for Storing CCD Images with openPMD (openPMD-CCD) v0.1.0

openPMD is an open meta-data schema that provides meaning and self-description to data sets in science and engineering. The openPMD-CCD software module adds interfaces to organize camera (CCD) images in hierarchical data files. This software provides modern I/O storage formats from high-performance computing and provides bindings for integration into experimental control systems, e.g. via Python 3 and LabView 2020.

Gonsalves, AnthonyJ↗
NASA NTRS1998

ADEOS Total Ozone Mapping Spectrometer (TOMS) Data Products User's Guide

Two data products from the Total Ozone Mapping Spectrometer (ADEOS/TOMS) have been archived at the Distributed Active Archive Center, in the form of Hierarchical Data Format files. The ADEOS/ TOMS began taking measurements on September 11, 1996, and ended on June 29, 1997. The instrument measured backscattered Earth radiance and incoming solar irradiance; their ratio was used in ozone retrievals. Changes in the reflectivity of the solar diffuser used for the irradiance measurement were monitored using a carousel of three diffusers, each exposed to the degrading effects of solar irradiation at different rates. The algorithm to retrieve total column ozone compares measured Earth radiances at sets of three wavelengths with radiances calculated for different total ozone values, solar zenith angles, and optical paths. The initial error in the absolute scale for TOMS total ozone is 3 percent, the one standard deviation random error is 2 percent, and the drift is less than 0.5 percent over the 9-month data record. The Level 2 product contains the measured radiances, the derived total ozone amount, and reflectivity information for each scan position. The Level 3 product contains daily total ozone and reflectivity in a 1-degree latitude by 1.25 degrees longitude grid. The Level 3 files containing estimates of UVB at the Earth surface and tropospheric aerosol information will also be available. Detailed descriptions of both HDF data files and the CDROM product are provided.

Krueger, A.↗
NASA NTRS2017

Reusing JPSS Ground System Components to Process Aura Ozone Monitoring Instrument Science Products

New Earth observation instruments are planned to enable advancements in Earth science research over the next decade. Diversity of Earth observing instruments and their observing platforms will continue to increase as new instrument technologies emerge and are deployed as part of National programs such as Joint Polar Satellite System (JPSS), Geostationary Operational Environmental Satellite system (GOES), Landsat as well as the potential for many CubeSat and aircraft missions. The practical use and value of these observational data often extends well beyond their original purpose. The practicing community needs intuitive and standardized tools to enable quick unfettered development of tailored products for specific applications and decision support systems. However, the associated data processing system can take years to develop and requires inherent knowledge and the ability to integrate increasingly diverse data types from multiple sources. This paper describes the adaptation of a large-scale data processing system built for supporting JPSS algorithm calibration and validation (CalVal) node to a simplified science data system for rapid application. The new configurable data system reuses scalable JAVA technologies built for the JPSS Government Resource for Algorithm Verification, Independent Test, and Evaluation (GRAVITE) system to run within a laptop environment and support product generation and data processing of AURA Ozone Monitoring Instrument (OMI) science products. Of particular interest are the root requirements necessary for integrating experimental algorithms and Hierarchical Data Format (HDF) data access libraries into a science data production system. This study demonstrates the ability to reuse existing Ground System technologies to support future missions with minimal changes.

Science Data Systems↗
NASA NTRS1998

Earth Probe Total Ozone Mapping Spectrometer (TOMS) Data Product User's Guide

Two data products from the Earth Probe Total Ozone Mapping Spectrometer (EP/TOMS) have been archived at the Distributed Active Archive Center, in the form of Hierarchical Data Format files. The EP/ TOMS began taking measurements on July 15, 1996. The instrument measures backscattered Earth radiance and incoming solar irradiance; their ratio is used in ozone retrievals. Changes in the reflectivity of the solar diffuser used for the irradiance measurement are monitored using a carousel of three diffusers, each exposed to the degrading effects of solar irradiation at different rates. The algorithm to retrieve total column ozone compares measured Earth radiances at sets of three wavelengths with radiances calculated for different total ozone values. The initial error in the absolute scale for TOMS total ozone is 3 percent, the one standard deviation random error is 2 percent, and the drift is less than 0.5 percent over the first year of data. The Level-2 product contains the measured radiances, the derived total ozone amount, and reflectivity information for each scan position. The Level-3 product contains daily total ozone and reflectivity in a 1-degree latitude by 1.25 degrees longitude grid. Level-3 files containing estimates of LTVB at the Earth surface and tropospheric aerosol information are also available, Detailed descriptions of both HDF data-files and the CD-ROM product are provided.

McPeters, R.↗
NASA NTRS2004

Indentured Parts List Maintenance and Part Assembly Capture Tool - IMPACT

Johnson Space Center's (JSC's) indentured parts list (IPL) maintenance and parts assembly capture tool (IMPACT) is an easy-to-use graphical interface for viewing and maintaining the complex assembly hierarchies of large databases. IMPACT, already in use at JSC to support the International Space Station (ISS), queries, updates, modifies, and views data in IPL and associated resource data, functions that it can also perform, with modification, for any large commercial database. By enabling its users to efficiently view and manipulate IPL hierarchical data, IMPACT performs a function unlike that of any other tool. Through IMPACT, users will achieve results quickly, efficiently, and cost effectively.

Jain, Bobby↗
NASA NTRSDate not supplied

Cloud Optimized Data Formats

Cloud computing offers the promise of being able to analyze Big Data earth Observations at scale, by allowing scientists to deploy many nodes at once to analyze the data. However, in order to take full advantage of cloud scalability, it is often necessary to reorganize and reformat the data to enable fine-grained, parallel access to the data in Web Object Storage. NASA recently conducted a study of several formats that are optimized for analysis in the cloud: Parquet, zarr, HDF (Hierarchical Data Format) in the Cloud, and Cloud-Optimized GeoTIFF (Tagged Image File Format). They were compared against non-cloud-optimized formats, netCDF (network Common Data Form) and GeoTIFF, with criteria based both on stewardship and analysis performance.

Christopher Lynnes↗
NASA NTRS1996

Nimbus-7 Total Ozone Mapping Spectrometer (TOMS) Data Products User's Guide

Two data products from the Total Ozone Mapping Spectrometer (TOMS) onboard Nimbus-7 have been archived at the Distributed Active Archive Center, in the form of Hierarchical Data Format files. The instrument measures backscattered Earth radiance and incoming solar irradiance; their ratio is used in ozone retrievals. Changes in the instrument sensitivity are monitored by a spectral discrimination technique using measurements of the intrinsically stable wavelength dependence of derived surface reflectivity. The algorithm to retrieve total column ozone compares measured Earth radiances at sets of three wavelengths with radiances calculated for different total ozone values, solar zenith angles, and optical paths. The initial error in the absolute scale for TOMS total ozone is 3 percent, the one standard deviation random error is 2 percent, and drift is less than 1.0 percent per decade. The Level-2 product contains the measured radiances, the derived total ozone amount, and reflectivity information for each scan position. The Level-3 product contains daily total ozone amount and reflectivity in a I - degree latitude by 1.25 degrees longitude grid. The Level-3 product also is available on CD-ROM. Detailed descriptions of both HDF data files and the CD-ROM product are provided.

McPeters, Richard D.↗
NASA NTRS2006

Using Heaps in Recursive Hierarchical Segmentation of Data

A modification to increase processing speed has been made in the algorithm and implementing software reported in Modified Recursive Hierarchical Segmentation of Data (GSC-14681-1), NASA Tech Briefs, Vol. 30, No. 6 (June 2006), page 51. That software performs recursive hierarchical segmentation of data having spatial characteristics (e.g., spectral-image data). The segmentation process includes an iterative subprocess, in each iteration of which it is necessary to determine a best pair of regions to merge [merges being justified by one or more measure(s) similarity of pixels in the regions]. In the previously reported version of the algorithm and software, the choice of a best pair of regions to merge involved the use of a fully sorted list of regions. That version was computationally inefficient because a fully sorted list is not needed: what is needed is only the identity of the pair of regions characterized by the smallest measure of dissimilarity. The present modification replaces the use of a fully sorted list with the use of data heaps, which are computationally more efficient for performing the required comparisons among dissimilarity measures. The modification includes the incorporation of standard and modified functions for creating and updating data heaps

Tilton, James C.↗
NASA NTRS2017

Data are from Mars, Tools are from Venus

Although during the data production phase, the data producers will usually ensure the products to be easily used by the specific power users the products serve. However, most data products are also posted for general public to use. It is not straightforward for data producers to anticipate what tools that these general end-data users are likely to use. In this talk, we will try to help fill in the gap by going over various tools related to Earth Science and how they work with the existing NASA HDF (Hierarchical Data Format) data products and the reasons why some products cannot be visualized or analyzed by existing tools. One goal is for to give insights for data producers on how to make their data product more interoperable. On the other hand, we also provide some hints for end users on how to make tools work with existing HDF data products. (tool category list: check the comments) HDF-EOS tools: HDFView HDF-EOS Plugin, HEG, h4tonccf, hdf-eos2 dumper, NCL, MATLAB, IDL, etc.net; CDF-Java tools: Panoply, IDV, toosUI, NcML, etc.net; CDF-C tools: ArcGIS Desktop, GrADS, NCL, NCO, etc.; GDAL tools: ArcGIS Desktop, QGIS, Google Earth, etc.; CSV tools: ArcGIS Online, MS Excel, Tableau, etc.

hdf↗
DOE OSTI2023

Counter Unmanned Aircraft System Metrics Tool

SAND2023-05387O The Counter Unmanned Aircraft System (CUAS) Metrics Tool consists of three applications: • The Data Collection Tool is designed to be used during testing and captures details that can be imported into the Figure Creator application. • The Unmanned Aircraft System (UAS) Log Converter converts different UAS logs into a standardized hierarchical data format with a timeline of geospatial data. • The Figure Creator develops relevant images for the test event. The application includes logic to ingest and correlate data from the Data Collection Tool and the UAS Log Converter to produce the chosen figures. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Mayle, Ashley↗
NASA NTRS1992

The ADAM environment and transputers

The ADAM environment is both used for data analysis by Starlink and for data acquisition by the UK-involved observatories in Australia, Hawaii, and the Canary Islands. ADAM was originally hosted under VAX/VMS but is now at an advanced stage of a Unix port. ADAM comprises a parameter system, hierarchical data system, noticeboard system, error handling system, and other components. Originally a multi-tasking single processor environment, it has been enhanced to a multiprocessor environment using local or wide area networking. The Royal Observatory Edinburgh is producing a transputer version of the ADAM kernel to allow instruments which make use of transputers for data acquisition/control to integrate more closely with the ADAM software running at the telescopes. Communication into the transputer system is based on Ethernet carrying TCP/IP, which eases development toward a network of mixed VMS/Unix/transputer Telescope systems. The transputer system is being applied to instruments under development for the UKIRT and JCMT telescopes.

Kelly, B. D.↗
NASA NTRSDate not supplied

High Performance Access to Archival Data Stored in HDF4 and HDF5 on Cloud Object Stores Without Reformatting the Files

Cloud computing offers numerous advantages for users of extensive Earth science data collections. These benefits encompass direct online access to data files and granules from any location, scalable access supporting parallel computing workflows, and flexible computing tools enabling innovative experimentation with processing techniques. However, older archival file formats designed for distinct computing systems hinder efficient access to decade-long time-series data when compared to data stored in modern cloud-optimized formats like Web Object Stores (WOS), exemplified by Amazon Web Services’ Simple Storage Service (S3). We describe DMR++ (Dataset Metadata Response plus plus), a technology facilitating efficient access to HDF5 (Hierarchical Data Format, version 5) and HDF4 files stored on WOS systems without requiring data reformatting. DMR++ achieves performance comparable to technologies like Zarr while preserving the original file structure, a substantial benefit considering the vast quantity of archival files held by organizations such as NASA. Moreover, DMR++ typically outperforms cloud-optimized versions of HDF5. Essentially an XML (Extensible Markup Language) document usually stored alongside the described data, DMR++ can also be generated on-the-fly but is generally created during data staging to the WOS. Archival files that use HDF4/5 often store large arrays of numerical data. The data in these files is often compressed, typically reducing their size by a factor of four or more. To achieve efficient access to portions of those arrays, they are 'chunked' into smaller sub-arrays, each individually compressed. The chunk size is a compromise, where spinning disks can efficiently access data in smaller chunks while S3 favors larger chunks. A simple optimization of aggregating smaller chunks that are stored adjacently, transferring them in a single access and then individually decompressing them will improve performance. NASA data pose an additional challenge: special Application Programmer Interface (API) libraries are often needed to compute some variables. These libraries are incompatible with WOS environments. Our solution involves storing computed values in the DMR++ document or a companion file, making them accessible like other variables and eliminating the need for specialized APIs. We outline specific optimizations for both satellite grid and swath data stored in HDF4-EOS2 (Earth Observing System).

James Gallagher↗
NASA NTRS2003

Exploring New Methods of Displaying Bit-Level Quality and Other Flags for MODIS Data

The NASA Distributed Active Archive Center (DAAC) at the National Snow and Ice Data Center (NSIDC) archives and distributes snow and sea ice products derived from the MODerate resolution Imaging Spectroradiometer (MODIS) on board NASA's Terra and Aqua satellites. All MODIS standard products are in the Earth Observing System version of the Hierarchal Data Format (HDF-EOS). The MODIS science team has packed a wealth of information into each HDF-EOS file. In addition to the science data arrays containing the geophysical product, there are often pixel-level Quality Assurance arrays which are important for understanding and interpreting the science data. Currently, researchers are limited in their ability to access and decode information stored as individual bits in many of the MODIS science products. Commercial and public domain utilities give users access, in varying degrees, to the elements inside MODIS HDF-EOS files. However, when attempting to visualize the data, users are confronted with the fact that many of the elements actually represent eight different 1-bit arrays packed into a single byte array. This project addressed the need for researchers to access bit-level information inside MODIS data files. In an previous NASA-funded project (ESDIS Prototype ID 50.0) we developed a visualization tool tailored to polar gridded HDF-EOS data set. This tool,called the Polar researchers to access, geolocate, visualize, and subset data that originate from different sources and have different spatial resolutions but which are placed on a common polar grid. The bit-level visualization function developed under this project was added to PHDIS, resulting in a versatile tool that serves a variety of needs. We call this the EOS Imaging Tool.

Khalsa, Siri Jodha Singh↗