Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “subsetting archiving”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Ultraviolet Signatures of Stars and Gas in Starburst Galaxies

We extracted about 500 short wavelength prime (SWP) (high-dispersion) International Ultraviolet Explorer (IUE) spectra of B0 to B8 stars of all luminosity classes for the purpose of creating a stellar library. Several hundred scientifically useful high-dispersion B-star spectra are in the IUE Archive. The data were taken either with the purpose of performing stellar-wind studies or investigations of interstellar absorption lines. Due to the nature of these projects, the S/N of the data is the maximum achievable with the IUE satellite. Rountree and Sonneborn and Walborn, Parker, and Nichols extracted a subset of B spectral classification standards from the IUE Archive to construct atlases of IUE high-dispersion spectra. Their work served as our guideline for the data extraction.

Leitherer, Claus↗

Framework for custom event sample augmentations for ATLAS analysis data

For HEP event processing, data is typically stored in column-wise synchronized containers, such as most prominently ROOT’s TTree, which have been used for several decades to store by now over 1 exabyte. These containers can combine row-wise association capabilities needed by most HEP event processing frameworks (e.g. Athena for ATLAS) with column-wise storage, which typically results in better compression and more efficient support for many analysis use-cases. One disadvantage is that these containers, TTree in the HEP use-case, require to contain the same attributes for each entry/row (representing events), which can make extending the list of attributes very costly in storage, even if those are only required for a small subsample of events. Since the initial design, the ATLAS software framework features powerful navigational infrastructure to allow storing custom data extensions for subsamples of events in separate, but synchronized containers. This allows adding event augmentations to ATLAS standard data products (such as DAOD-PHYS or PHYSLITE) avoiding duplication of those core data products, while limiting their size increase. For this functionality, the framework does not rely on any associations made by the I/O technology (i.e. ROOT), however it supports TTree friends and builds the associated index to allow for analysis outside of the ATLAS framework. A prototype based on the Long-Lived Particle search is implemented and preliminary results with this prototype will be presented. At this point, augmented data are stored within the same file as the core data. Storing them in separate files will be investigated in future, as this could provide more flexibility, e.g. certain sites may only want a subset of several augmentations or augmentations can be archived to tape once their analysis is complete.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

ERA5-Land Data for LASSO-CACTI Overview Paper

The European Centre for Medium-Range Weather Forecasts (ECMWF) generated a soil reanalysis dataset for the land component of the fifth generation of European ReAnalysis (ERA5), referred to as ERA5-Land. This is a model-generated dataset, with the original version available for the period 1950 to present. The version archived in this DOE ARM product is a subset of the data is for the period of the CACTI field campaign plus several preceding months, specifically from August 1, 2018 through March 22, 2019 with hourly intervals. The ARM copy is also a sub-region of the original global product; the ARM copy is for -60 to -5 °N by -105 to -30 °W. Only variables necessary to drive the WRF-Hydro model are included, which are the 2-m temperature and specific humidity, 10-m wind components, surface pressure, rain rate, and downward surface short and longwave radiation. These data have been obtained from the Copernicus Data Store.

10m wind u-component↗

NASA’s Atmospheric Science Data Center’s Approach to a Cloud-Based Model of Ingest, Archival, and Distribution of TEMPO Data: Methods, Challenges, and Best Practices

The National Aeronautics and Space Administration's (NASA) Atmospheric Science Data Center (ASDC) at NASA Langley Research Center in Hampton, VA provides atmospheric science data products and services to the science community, including enhanced search and subsetting capabilities for numerous datasets. The ASDC is the official Distributed Active Archive Center (DAAC) of record for the upcoming Tropospheric Emissions: Monitoring of Pollution (TEMPO) instrument. TEMPO will be situated on a geostationary satellite positioned at a longitude near the center of the conterminous United States and focused on North America, making hourly swaths of its field of regard from east to west. ASDC’s data products are currently hosted locally and services (e.g., spatial and temporal subsetting) are managed on premises. The ASDC is planning to provide TEMPO data and services in the cloud through the Earthdata Search platform. This presentation will discuss the ASDC’s approach to a cloud-based model of ingest, archival, and distribution of TEMPO data. Methods, challenges, best practices, lessons learned, and future plans will be discussed.

Iman Nasif↗

Global analysis of Drosophila Cys2-His2 zinc finger proteins reveals a multitude of novel recognition motifs and binding determinants

Cys2-His2 zinc finger proteins (ZFPs) are the largest group of transcription factors in higher metazoans. A complete characterization of these ZFPs and their associated target sequences is pivotal to fully annotate transcriptional regulatory networks in metazoan genomes. As a first step in this process, we have characterized the DNA-binding specificities of 129 zinc finger sets from Drosophila using a bacterial one-hybrid system. This data set contains the DNA-binding specificities for at least one encoded ZFP from 70 unique genes and 23 alternate splice isoforms representing the largest set of characterized ZFPs from any organism described to date. These recognition motifs can be used to predict genomic binding sites for these factors within the fruit fly genome. Subsets of fingers from these ZFPs were characterized to define their orientation and register on their recognition sequences, thereby allowing us to define the recognition diversity within this finger set. We find that the characterized fingers can specify 47 of the 64 possible DNA triplets. To confirm the utility of our finger recognition models, we employed subsets of Drosophila fingers in combination with an existing archive of artificial zinc finger modules to create ZFPs with novel DNA-binding specificity. These hybrids of natural and artificial fingers can be used to create functional zinc finger nucleases for editing vertebrate genomes.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Archive data base and handling system for the Orbiter flying qualities experiment program

The OFQ archives data base and handling system assembled as part of the Orbiter Flying Qualities (OFQ) research of the Orbiter Experiments Program (EOX) are described. The purpose of the OFQ archives is to preserve and document shuttle flight data relevant to vehicle dynamics, flight control, and flying qualities in a form that permits maximum use for qualified users. In their complete form, the OFQ archives contain descriptive text (general information about the flight, signal descriptions and units) as well as numerical time history data. Since the shuttle program is so complex, the official data base contains thousands of signals and very complex entries are required to obtain data. The OFQ archives are intended to provide flight phase oriented data subsets with relevant signals which are easily identified for flying qualities research.

Myers, T. T.↗

Satellite data management for effective data access

The management of data generated from satellite missions has not always led to effective access of that data by the scientific community. NASA has tried to alleviate this problem for ocean scientists, by initiating a program, the NASA Ocean Data System (NODS). The menu-based user interface that NODS employs allows a user to make request and receive answers within a short time of accessing the system. A catalog system, which holds information about oceanographic data sets may be queried to determine the suitability of a particular data set. Once a candidate data set is found, the user is directed to the person or place which actually holds the data. NODS also has an archive system that holds data from ocean-observing satellites. The archive may be queried to obtain a manageable data subset that can be delivered in a useful form.

Hogan, Patrick D.↗

INTRAGROUP AND GALAXY-LINKED DIFFUSE X-RAY EMISSION IN HICKSON COMPACT GROUPS

Isolated compact groups (CGs) of galaxies present a range of dynamical states, group velocity dispersions, and galaxy morphologies with which to study galaxy evolution, particularly the properties of gas both within the galaxies and in the intragroup medium. As part of a large, multiwavelength examination of CGs, we present an archival study of diffuse X-ray emission in a subset of nine Hickson compact groups (HCGs) observed with the Chandra X-Ray Observatory. We find that seven of the groups in our sample exhibit detectable diffuse emission. However, unlike large-scale emission in galaxy clusters, the diffuse features in the majority of the detected groups are linked to the individual galaxies, in the form of both plumes and halos likely as a result of vigourous star formation or activity in the galaxy nucleus, as well as in emission from tidal features. Unlike previous studies from earlier X-ray missions, HCGs 31, 42, 59, and 92 are found to be consistent with the L{sub X} -T relationship from clusters within the errors, while HCGs 16 and 31 are consistent with the cluster L{sub X} -{sigma} relation, though this is likely coincidental given that the hot gas in these two systems is largely due to star formation. We find that L{sub X} increases with decreasing group H I to dynamical-mass ratio with tentative evidence for a dependence in X-ray luminosity on H I morphology whereby systems with intragroup H I indicative of strong interactions are considerably more X-ray luminous than passively evolving groups. We also find a gap in the L{sub X} of groups as a function of the total group specific star formation rate. Our findings suggest that the hot gas in these groups is not in hydrostatic equilibrium and these systems are not low-mass analogs of rich groups or clusters, with the possible exception of HCG 62.

79 ASTRONOMY AND ASTROPHYSICS↗

TOPEX/POSEIDON GPS Demonstration Experiment Data

Global Positioning Satellite (GPS) measurements obtained from the on-board TOPEX/POSEIDON demonstration flight receiver and a global network of ground receivers are available from the GPS Data Handling Facility (GDHF) archive. Ground receiver observations are derived from a six station subset of the GPS global tracking network. The six TOPEX ground stations yielded 95 percent of the expected data during the six month demonstration phase from November, 1992 to April, 1993. For the same period, 98 percent of the available P-code tracking from the TOPEX flight receiver was obtained. Raw carrier phase and P-code pseudorange measurements are provided for the ground and flight receivers along with an additional file of calibrated and compressed flight receiver data...

Guinn, J. R.↗

Intragroup and Galaxy-linked Diffuse X-ray Emission In Hickson Compact Groups

Isolated compact groups (CGs) of galaxies present a range of dynamical states, group velocity dispersions, and galaxy morphologies with which to study galaxy evolution, particularly the properties of gas both within the galaxies and in the intragroup medium. As part of a large, multiwavelength examination of CGs, we present an archival study of diffuse X-ray emission in a subset of nine Hickson compact groups (HCGs) observed with the Chandra X-Ray Observatory. We find that seven of the groups in our sample exhibit detectable diffuse emission. However, unlike large-scale emission in galaxy clusters, the diffuse features in the majority of the detected groups are linked to the individual galaxies, in the form of both plumes and halos likely as a result of vigourous star formation or activity in the galaxy nucleus, as well as in emission from tidal features. Unlike previous studies from earlier X-ray missions, HCGs 31, 42, 59, and 92 are found to be consistent with the L(sub X-Tau) relationship from clusters within the errors, while HCGs 16 and 31 are consistent with the cluster L(sub X-sigma) relation, though this is likely coincidental given that the hot gas in these two systems is largely due to star formation. We find that L(sub X) increases with decreasing group Hi to dynamical-mass ratio with tentative evidence for a dependence in X-ray luminosity on Hi morphology whereby systems with intragroup Hi indicative of strong interactions are considerably more X-ray luminous than passively evolving groups. We also find a gap in the L(sub X) of groups as a function of the total group specific star formation rate. Our findings suggest that the hot gas in these groups is not in hydrostatic equilibrium and these systems are not low-mass analogs of rich groups or clusters, with the possible exception of HCG 62.

Desjardins, Tyler D.↗

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly.

Lynnes, Christopher↗

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly. Reviewed by Mark McInerney ESDIS Deputy Project Manager.

Lynnes, Christopher↗

Use of data description languages in the interchange of data

The Consultative Committee for Space Data Systems (CCSDS) is developing Standards for the interchange of information between systems, including those operating under different environments. The objective is to perform the interchange automatically, i.e. in a computer interpretable manner. One aspect of the concept developed by CCSDS is the use of a separate data description to specify the data being transferred. Using the description, data can then be automatically parsed by the receiving computer. With a suitably expressive Data Description Language (DDL), data formats of arbitrary complexity can be handled. The advantages of this approach are: (1) that the description need only be written and distributed once to all users, and (2) new software does not need to be written for each new format, provided generic tools are available to support writing and interpretation of descriptions and the associated data instances. Consequently, the effort of 'hard coding' each new format is avoided and problems of integrating multiple implementations of a given format by different users are avoided. The approach is applicable in any context where computer parsable description of data could enhance efficiency (e.g. within a spacecraft control system, a data delivery system or an archive). The CCSDS have identified several candidate DDL's: EAST (Extended Ada Subset), TSDN (Transfer Syntax Data Notation) and MADEL (Modified ASN.1 as a Data Description Language -- a DDL based on the Abstract Syntax Notation One - ASN.1 - specified in the ISO/IEC 8824). This paper concentrates on ESA's development of MADEL. ESA have also developed a 'proof of concept' prototype of the required support tools, implemented on a PC under MS-DOS, which has successfully demonstrated the feasibility of the approach, including the capability within an application of retrieving and displaying particular data elements, given its MADEL description (i.e. a data description written in MADEL). This paper outlines the work done to date and assesses the applicability of this modified ASN.1 as a DDL. The feasibility of the approach is illustrated with several examples.

Pignede, M.↗

Community-Based Services that Facilitate Interoperability and Intercomparison of Precipitation Datasets from Multiple Sources

Over the past 12 years, large volumes of precipitation data have been generated from space-based observatories (e.g., TRMM), merging of data products (e.g., gridded 3B42), models (e.g., GMAO), climatologies (e.g., Chang SSM/I derived rain indices), field campaigns, and ground-based measuring stations. The science research, applications, and education communities have greatly benefited from the unrestricted availability of these data from the Goddard Earth Sciences Data and Information Services Center (GES DISC) and, in particular, the services tailored toward precipitation data access and usability. In addition, tools and services that are responsive to the expressed evolving needs of the precipitation data user communities have been developed at the Precipitation Data and Information Services Center (PDISC) (http://disc.gsfc.nasa.gov/precipitation or google NASA PDISC), located at the GES DISC, to provide users with quick data exploration and access capabilities. In recent years, data management and access services have become increasingly sophisticated, such that they now afford researchers, particularly those interested in multi-data set science analysis and/or data validation, the ability to homogenize data sets, in order to apply multi-variant, comparison, and evaluation functions. Included in these services is the ability to capture data quality and data provenance. These interoperability services can be directly applied to future data sets, such as those from the Global Precipitation Measurement (GPM) mission. This presentation describes the data sets and services at the PDISC that are currently used by precipitation science and applications researchers, and which will be enhanced in preparation for GPM and associated multi-sensor data research. Specifically, the GES-DISC Interactive Online Visualization ANd aNalysis Infrastructure (Giovanni) will be illustrated. Giovanni enables scientific exploration of Earth science data without researchers having to perform the complicated data access and match-up processes. In addition, PDISC tool and service capabilities being adapted for GPM data will be described, including the Google-like Mirador data search and access engine; semantic technology to help manage large amounts of multi-sensor data and their relationships; data access through various Web services (e.g., OPeNDAP, GDS, WMS, WCS); conversion to various formats (e.g., netCDF, HDF, KML (for Google Earth)); visualization and analysis of Level 2 data profiles and maps; parameter and spatial subsetting; time and temporal aggregation; regridding; data version control and provenance; continuous archive verification; and expertise in data-related standards and interoperability. The goal of providing these services is to further the progress towards a common framework by which data analysis/validation can be more easily accomplished.

Liu, Zhong↗

The CCD/Transit Instrument (CTI) data-analysis system

The automated software system for archiving, analyzing, and interrogating data from the CCD/Transit Instrument (CTI) is described. The CTI collects up to 450 Mbytes of image-data each clear night in the form of a narrow strip of sky observed in two colors. The large data-volumes and the scientific aims of the project make it imperative that the data are analyzed within the 24-hour period following the observations. To this end a fully automatic and self evaluating software system has been developed. The data are collected from the telescope in real-time and then transported to Tucson for analysis. Verification is performed by visual inspection of random subsets of the data and obvious cosmic rays are detected and removed before permanent archival is made to the optical disc. The analysis phase is performed by a pair of linked algorithms, one operating on the absolute pixel-values and the other on the spatial derivative of the data. In this way both isolated and merged images are reliably detected in a single pass. In order to isolate the latter algorithm from the effects of noise spikes a 3x3 Hanning filter is applied to the raw data before the analysis is run. The algorithms reduce the input pixel-data to a database of measured parameters for each image which has been found. A contrast filter is applied in order to assign a detection-probability to each image and then x-y calibration and intensity calibration are performed using known reference stars in the strip. These are added to as necessary by secondary standards boot-strapped from the CTI data itself. The final stages involve merging the new data into the CTI Master-list and History-list and the automatic comparison of each new detection with a set of pre-defined templates in parameter-space to find interesting objects such as supernovae, quasars and variable stars. Each stage of the processing from verification to interesting image selection is performed under a data-logging system which both controls the pipe-lining of data through the system and records key performance monitor parameters which are built into the software. Furthermore, the data from each stage are stored in databases to facilitate evaluation, and all stages offer the facility to enter keyword-indexed free-format text into the data-logging system. In this way a large measure of certification is built into the system to provide the necessary confidence in the end results.

Cawson, M. G. M.↗

The NSO FTS database program and archive (FTSDBM)

Data from the NSO Fourier transform spectrometer is being re-archived from half inch tape onto write-once compact disk. In the process, information about each spectrum and a low resolution copy of each spectrum is being saved into an on-line database. FTSDBM is a simple database management program in the NSO external package for IRAF. A command language allows the FTSDBM user to add entries to the database, delete entries, select subsets from the database based on keyword values including ranges of values, create new database files based on these subsets, make keyword lists, examine low resolution spectra graphically, and make disk number/file number lists. Once the archive is complete, FTSDBM will allow the database to be efficiently searched for data of interest to the user and the compact disk format will allow random access to that data.

Lytle, D. M.↗

Development of an Improved Spatial Metadata Simplification Algorithm

The National Aeronautics and Space Administration's (NASA) Atmospheric Science Data Center (ASDC) at NASA Langley Research Center in Hampton, VA provides atmospheric science data products and services to the science community, including enhanced search and subsetting capabilities for numerous Earth Science datasets. The ASDC is the official Distributed Active Archive Center (DAAC) of record for the Tropospheric Emissions: Monitoring of Pollution (TEMPO) instrument. TEMPO is situated on a geostationary satellite positioned at a longitude near the center of the conterminous United States and focused on North America, making hourly swaths of its field of regard from east to west. Spatial metadata is an essential component for the discovery and distribution of Earth Science data. The simplified polygonal boundaries representing the archived data files ensure that any granule can be identified quickly and accurately by a geospatial query. Historically the Douglas-Peucker algorithm has been used for polygon simplification; however, due to the nature of the algorithm, a buffer must be added to the polygon before simplification to ensure pivotal points are not removed by the algorithm. This adds in additional error to the polygon simplification. ASDC’s goal is to test other methods of polyline simplification, such as Visvalingan-Whyatt and Opheim simplification alongside of Douglas-Peucker and different buffering methods, to produce less error during polygon simplification of TEMPO data swaths, and special spatial query geometries such as EPA non-attainment regions, and geopolitical boundaries.

Spatial Metadata↗

Use of Spatial Metadata Simplification for TEMPO

The National Aeronautics and Space Administration's (NASA) Atmospheric Science Data Center (ASDC) at NASA Langley Research Center in Hampton, VA provides atmospheric science data products and services to the science community, including enhanced search and subsetting capabilities for numerous Earth Science datasets. The ASDC is the official Distributed Active Archive Center (DAAC) of record for the Tropospheric Emissions: Monitoring of Pollution (TEMPO) instrument. TEMPO is situated on a geostationary satellite positioned at a longitude near the center of the conterminous United States and focused on North America, making hourly swaths of its field of regard from east to west. Spatial metadata is an essential component for the discovery and distribution of Earth Science data. The simplified polygonal boundaries representing the archived data files ensure that any granule can be identified quickly and accurately by a geospatial query. Historically the Douglas-Peucker algorithm has been used for polygon simplification; however, due to the nature of the algorithm, a buffer must be added to the polygon before simplification to ensure pivotal points are not removed by the algorithm. This adds in additional error to the polygon simplification. ASDC’s goal is to test other methods of polyline simplification, such as Visvalingan-Whyatt and Opheim simplification alongside of Douglas-Peucker and different buffering methods, to produce less error during polygon simplification of TEMPO data swaths, and special spatial query geometries such as EPA non-attainment regions, and geopolitical boundaries.

Spatial Metadata↗