Merged and Gridded GPM and Atmospheric River Data Product
Explore the source record for details and available documents.
Marian E. Mateling↗
SEARCH · Engineering Papers
Results for “production data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Enhancing the IGS Data and Products Infrastructure: A Data Center Perspective
The Data Transfer and Data Centers session will present papers on current data center activities as well as various topics geared towards enhancing the availability and acquisition of International Global Positions System Service (IGS) data and products. These topics include: strategies for reducing data file latency at IGS Data Centers, standardization of data file replacement notification methods, improving the efficiency of data file transfer among IGS Data Centers, and defining a network topology for GPS data files for use in applications.
Scharber, Michael↗
Calibration Refinements in Support of MISR
The Multi-angle Imaging SpectroRadiometer (MISR) is one of five instruments on-board the EOSI Terra spacecraft, and has collected science data since March 2000. A multi-angle capability is provided by ninecameras, which view up to 70' forward and aft of the spacecraft track and enable unique geophysical retrievals. Throughout its mission, a calibration team has made periodic refinements to the process used to calibrate MISR. These have resulted in improved absolute, and band and camera-relative calibrations, as well as in derived geophysical data products. Data reprocessing is on-going, such that these refinements also improve previous data acquisitions. The calibration process is believed to be mature at this time, with no other changes anticipated. Bimonthly deployments of the on-board-calibrator continue to monitor instrument response degradations, and provide correction coefficients needed to maintain the accuracy of the radiance products.
refinements↗
Landsat missions
This paper gives a description and present status of NASA's Landsat program, including the orbital coverage, payload, data processing, and data products. Data uses are discussed in general terms. Also discussed are NASA's plans for the Landsat-3 and Landsat-D missions and the improved capabilities represented by these missions. The Shuttle retrieval of Landsat-D and the Shuttle launch of future Landsat satellites are discussed conceptually.
Mannheimer, H.↗
The moderate resolution imaging spectrometer data system
Science requirements for a moderate resolution imaging spectrometer (MODIS) data system are outlined, as well as the operations concept and system specifications based on these requirements. Two systems are considered - MODIS-N for a 40-channel nadir-viewing scanner and MODIS-T for a 64-channel tiltable scanner. The first system collects data from 15 thermal-infrared channels at all times and from 25 reflected-energy channels during daytime, while MODIS-T will take data from 64 channels on a 100-percent duty cycle during daytime only. Standard data products, data acquisition and processing, and requirements on data are discussed. Emphasis is placed on the interfaces between the science data processing facilities and the scientific users.
Han, Daesoo↗
A Metadata Action Language
The data management problem comprises data processing and data tracking. Data processing is the creation of new data based on existing data sources. Data tracking consists of storing metadata descriptions of available data. This paper addresses the data management problem by casting it as an AI planning problem. Actions are data-processing commands, plans are dataflow programs and goals are metadata descriptions of desired data products. Data manipulation is simply plan generation and execution, and a key component of data tracking is inferring the effects of an observed plan. We introduce a new action language for data management domains, called ADILM. We discuss the connection between data processing and information integration and show how a language for the latter must be modified to support the former. The paper also discusses information gathering within a data-processing framework, and show how ADILM metadata expressions are a generalization of Local Completeness.
Golden, Keith↗
Automated Data Processing as an AI Planning Problem
NASA s vision for Earth Science is to build a "sensor web"; an adaptive array of heterogeneous satellites and other sensors that will track important events, such as storms, and provide real-time information about the state of the Earth to a wide variety of customers. Achieving his vision will require automation not only in the scheduling of the observations but also in the processing af tee resulting data. Ta address this need, we have developed a planner-based agent to automatically generate and execute data-flow programs to produce the requested data products. Data processing domains are substantially different from other planning domains that have been explored, and this has led us to substantially different choices in terms of representation and algorithms. We discuss some of these differences and discuss the approach we have adopted.
Golden, Keith↗
Addressing the Big-Earth-Data Variety Challenge with the Hierarchical Triangular Mesh
We have implemented an updated Hierarchical Triangular Mesh (HTM) as the basis for a unified data model and an indexing scheme for geoscience data to address the variety challenge of Big Earth Data. We observe that, in the absence of variety, the volume challenge of Big Data is relatively easily addressable with parallel processing. The more important challenge in achieving optimal value with a Big Data solution for Earth Science (ES) data analysis, however, is being able to achieve good scalability with variety. With HTM unifying at least the three popular data models, i.e. Grid, Swath, and Point, used by current ES data products, data preparation time for integrative analysis of diverse datasets can be drastically reduced and better variety scaling can be achieved. In addition, since HTM is also an indexing scheme, when it is used to index all ES datasets, data placement alignment (or co-location) on the shared nothing architecture, which most Big Data systems are based on, is guaranteed and better performance is ensured. Moreover, our updated HTM encoding turns most geospatial set operations into integer interval operations, gaining further performance advantages.
SciDB↗
Ocean Surface Vector Wind: Research Challenges and Operational Opportunities
The atmosphere and ocean are joined together over seventy percent of Earth, with ocean surface vector wind (OSVW) stress one of the linkages. Satellite OSVW measurements provide estimates of wind divergence at the bottom of the atmosphere and wind stress curl at the top of the ocean; both variables are critical for weather and climate applications. As is common with satellite measurements, a multitude of OSVW data products exist for each currently operating satellite instrument. In 2012 the Joint Technical Commission on Oceanography and Marine Meteorology (JCOMM) launched an initiative to coordinate production of OSVW data products to maximize the impact and benefit of existing and future OSVW measurements in atmospheric and oceanic applications. This paper describes meteorological and oceanographic requirements for OSVW data products; provides an inventory of unique data products to illustrate that the challenge is not the production of individual data products, but the generation of harmonized datasets for analysis and synthesis of the ensemble of data products; and outlines a vision for JCOMM, in partnership with other international groups, to assemble an international network to share ideas, data, tools, strategies, and deliverables to improve utilization of satellite OSVW data products for research and operational applications.
Joint Technical Commission on Oceanography and Mar↗
The Development of Two Science Investigator-led Processing Systems (SIPS) for NASA's Earth Observation System (EOS)
In 2001, NASA Goddard Space Flight Center's Laboratory for Terrestrial Physics started the construction of a science Investigator-led Processing System (SIPS) for processing data from the Ozone Monitoring Instrument (OMI) which will launch on the Aura platform in mid 2004. The Ozone Monitoring Instrument (OMI) is a contribution of the Netherlands Agency for Aerospace Programs (NIVR) in collaboration with the Finnish Meteorological Institute (FMI) to the Earth Observing System (EOS) Aura mission. It will continue the Total Ozone Monitoring System (TOMS) record for total ozone and other atmospheric parameters related to ozone chemistry and climate. OMI measurements will be highly synergistic with the other instruments on the EOS Aura platform. The LTP previously developed the Moderate Resolution Imaging Spectrometer (MODIS) Data Processing System (MODAPS), which has been in full operations since the launches of the Terra and Aqua spacecrafts in December, 1999 and May, 2002 respectively. During that time, it has continually evolved to better support the needs of the MODIS team. We now run multiple instances of the system managing faster than real time reprocessings of the data as well as continuing forward processing. The new OMI Data Processing System (OMIDAPS) was adapted from the MODAPS. It will ingest raw data from the satellite ground station and process it to produce calibrated, geolocated higher level data products. These data products will be transmitted to the Goddard Distributed Active Archive Center (GDAAC) instance of the Earth Observing System (EOS) Data and Information System (EOSDIS) for long term archive and distribution to the public. The OMIDAPS will also provide data distribution to the OMI Science Team for quality assessment, algorithm improvement, calibration, etc. We have taken advantage of lessons learned from the MODIS experience and software already developed for MODIS. We made some changes in the hardware system organization, database and software to adapt the system for OMI. We replaced the fundamental database system, Sybase, with an Open Source RDBMS called PostgreSQL, and based the entire OMIDAPS on a cluster of Linux based commodity computers rather than the large SGI servers that MODAPS uses. Rather than relying on a central I/O server host, the new system distributes its data archive among multiple server hosts in the cluster. OMI is also customizing the graphical user interfaces and reporting structure to more closely meet the needs of the OMI Science Team. Prior to 2003, simulated OMI data and the science algorithms were not ready for production testing. We initially constructed a prototype system and tested using a 25 year dataset of Total Ozone Mapping Spectrometer (TOMS) and Solar Backscatter Ultraviolet Instrument (SBUV) data. This prototype system provided a platform to support the adaptation of the algorithms for OMI, and provided reprocessing of the historical data aiding in its analysis. In a recent reanalysis of the TOMS data, the OMIDAPS processed 108,000 full orbits of data through 4 processing steps per orbit, producing about 800,000 files (400 GiB) of level 2 and greater data files. More recently we have installed two instances of the OMIDAPS for integration and testing of OM1 science processes as they get delivered from the Science Team. A Test instance of the OMIDAPS has also supported a series of "Interface Confidence Tests" (ICTs) and End-to-End Ground System tests to ensure the launch readiness of the system. This paper will discuss the high-level hardware, software, and database organization of the OMIDAPS and how it builds on the MODAPS heritage system. It will also provide an overview of the testing and implementation of the production OMIDAPS.
Tilmes, Curt↗
A User-Focused Renovation of CERES Metadata
Production software and public data products for Clouds and the Earth’s Radiant Energy System (CERES) continue to evolve as the project extends its climate data record. The data management team for CERES is currently undertaking major renovations of both code and data products, the latter of which is, of course, in service of improving user experience. A major mode of CERES’ data product improvement is in renovating products’ metadata. Metadata standards have evolved since CERES began producing its data products in 2000. In its twentieth year, CERES essentially asked the question: how would the project design its data products if it could start all over again? With forthcoming editions, this rebirth will be realized. CERES has redesigned its metadata standards to best position itself for data discoverability. The project has used the latest standards being developed in NASA’s Earth Science Data and Information Systems (ESDIS) Project’s Unified Metadata Model (UMM) documentation; collaborated with the Atmospheric Science Data Center (ASDC) to ensure compliance with Common Metadata Repository compatibility, and continued compliance with Climate and Forecast (CF) Conventions. In doing so, the team created its own, internal document for proper metadata creation and metadata verification software that is deployed prior to all code deliveries. This presentation will discuss this redesign process, as well as needs met and those that are still outstanding in the search for an improved user experience with CERES data products.
Kathleen Dejwakh↗
Processing Raw HST Data With Up-to-Date Calibration Data
On-the-Fly Reprocessing (OTFR) is a collection of data-processing routines that work within the context of the Hubble Space Telescope (HST) pipeline data-flow system. The purpose served by OTFR is to generate, on demand, scientifically useful data products from raw HST data stored in an archive. First, on the basis of the requested final data products, OTFR retrieves the corresponding sets of raw data from the archives. Next, OTFR processes the raw data sets to remove artifacts and to establish proper header and other template information. Finally, the calibration routines appropriate to the specific data sets are invoked to produce the requested data products, and the data products are released to an archive distribution system for transmission to the requesting party. OTFR offers two notable advantages: (1) Inasmuch as calibrated data occupy about 8 times as much storage space as do raw data, by obviating storage of calibrated data, OTFR reduces the storage capacity needed by the archive; and (2) the calibration routines can be updated to give requesters the benefit of the most recent calibrations.
Miller, Warren↗
TESS Data Release Notes: Sector 24, DR35
These Data Release Notes provide information on the processing and export of data from the Transiting Exoplanet Survey Satellite (TESS). The data products included in this data release are full frame images (FFIs), target pixel fi les, light curve fi les, collateral pixel fi les, cotrending basis vectors (CBVs), and Data Validation (DV) reports, time series, and associated xml fi les. These data products were generated by the TESS Science Processing Operations Center (SPOC, Jenkins et al., 2016) at NASA Ames Research Center from data collected by the TESS instrument, which is managed by the TESS Payload Operations Center (POC) at Massachusetts Institute of Technology (MIT). The format and content of these data products are documented in the Science Data Products Description Document (SDPDD). The SPOC science algorithms are based heavily on those of the Kepler Mission science pipeline, and are described in the Kepler Data Processing Handbook (Jenkins, 2019).2 The Data Validation algorithms are documented in Twicken et al. (2018) and Li et al. (2019). The TESS Instrument Handbook (Vanderspek et al., 2018) contains more information about the TESS instrument design, detector layout, data properties, and mission operations. The TESS Mission is funded by NASA's Science Mission Directorate.
TESS↗
TESS Data Release Notes: Sector 21 DR 29
These Data Release Notes provide information on the processing and export of data from the Transiting Exoplanet Survey Satellite (TESS). The data products included in this data release are full frame images (FFIs), target pixel les, light curve les, collateral pixel les, cotrending basis vectors (CBVs), and Data Validation (DV) reports, time series, and associated xml les. These data products were generated by the TESS Science Processing Operations Center (SPOC, Jenkins et al., 2016) at NASA Ames Research Center from data collected by the TESS instrument, which is managed by the TESS Payload Operations Center (POC) at Massachusetts Institute of Technology (MIT). The format and content of these data products are documented in the Science Data Products Description Document (SDPDD)1. The SPOC science algorithms are based heavily on those of the Kepler Mission science pipeline, and are described in the Kepler Data Processing Handbook (Jenkins, 2017).2 The Data Validation algorithms are documented in Twicken et al. (2018) and Li et al. (2019). The TESS Instrument Handbook (Vanderspek et al., 2018) contains more information about the TESS instrument design, detector layout, data properties, and mission operations. The TESS Mission is funded by NASA's Science Mission Directorate.
Michael M. Fausnaugh↗
Online Tools for Uncovering Data Quality (DQ) Issues in Satellite-Based Global Precipitation Products
Data quality (DQ) has many attributes or facets (i.e., errors, biases, systematic differences, uncertainties, benchmark, false trends, false alarm ratio, etc.)Sources can be complicated (measurements, environmental conditions, surface types, algorithms, etc.) and difficult to be identified especially for multi-sensor and multi-satellite products with bias correction (TMPA, IMERG, etc.) How to obtain DQ info fast and easily, especially quantified info in ROI Existing parameters (random error), literature, DIY, etc.How to apply the knowledge in research and applications.Here, we focus on online systems for integration of products and parameters, visualization and analysis as well as investigation and extraction of DQ information.
precipitation↗
Data are from Mars, Tools are from Venus
Although during the data production phase, the data producers will usually ensure the products to be easily used by the specific power users the products serve. However, most data products are also posted for general public to use. It is not straightforward for data producers to anticipate what tools that these general end-data users are likely to use. In this talk, we will try to help fill in the gap by going over various tools related to Earth Science and how they work with the existing NASA HDF (Hierarchical Data Format) data products and the reasons why some products cannot be visualized or analyzed by existing tools. One goal is for to give insights for data producers on how to make their data product more interoperable. On the other hand, we also provide some hints for end users on how to make tools work with existing HDF data products. (tool category list: check the comments) HDF-EOS tools: HDFView HDF-EOS Plugin, HEG, h4tonccf, hdf-eos2 dumper, NCL, MATLAB, IDL, etc.net; CDF-Java tools: Panoply, IDV, toosUI, NcML, etc.net; CDF-C tools: ArcGIS Desktop, GrADS, NCL, NCO, etc.; GDAL tools: ArcGIS Desktop, QGIS, Google Earth, etc.; CSV tools: ArcGIS Online, MS Excel, Tableau, etc.
hdf↗
ESIP Information Quality Cluster (IQC)
The Information Quality Cluster (IQC) within the Federation of Earth Science Information Partners (ESIP) was initially formed in 2011 and has evolved significantly over time. The current objectives of the IQC are to: 1. Actively evaluate community data quality best practices and standards; 2. Improve capture, description, discovery, and usability of information about data quality in Earth science data products; 3. Ensure producers of data products are aware of standards and best practices for conveying data quality, and data providers distributors intermediaries establish, improve and evolve mechanisms to assist users in discovering and understanding data quality information; and 4. Consistently provide guidance to data managers and stewards on how best to implement data quality standards and best practices to ensure and improve maturity of their data products. The activities of the IQC include: 1. Identification of additional needs for consistently capturing, describing, and conveying quality information through use case studies with broad and diverse applications; 2. Establishing and providing community-wide guidance on roles and responsibilities of key players and stakeholders including users and management; 3. Prototyping of conveying quality information to users in a more consistent, transparent, and digestible manner; 4. Establishing a baseline of standards and best practices for data quality; 5. Evaluating recommendations from NASA's DQWG in a broader context and proposing possible implementations; and 6. Engaging data providers, data managers, and data user communities as resources to improve our standards and best practices. Following the principles of openness of the ESIP Federation, IQC invites all individuals interested in improving capture, description, discovery, and usability of information about data quality in Earth science data products to participate in its activities.
data products↗
NASA EOSDIS Data Usage Metrics- Insight and Assessment
NASA's Earth Science Data and Information System (ESDIS) Project collects Earth science data usage metrics on a daily basis through the ESDIS Metrics System (EMS). This includes metrics on distribution of data products, users, data volumes, and number of files, which are key parameters in evaluating system-level performance of any of the Distributed Active Archive Centers (DAACs) encompassed by NASA's Earth Observing System (EOS) Data and Information System (EOSDIS). EOSDIS data usage metrics illustrate the benefits of making NASA data openly available to the public and show a rapid growth in data distribution to a worldwide user community. In fact, each year since 2014 the EOSDIS has distributed over one billion data files of products from EOS satellite, airborne, and in situ observations. An assessment of the long-term trends of data usage metrics and user characterization provides insights into data usability.This study will focus on describing the EMS as a metrics collection tool and will provide a comprehensive analysis of EOSDIS data usage metrics over the last 10 years. This study will also characterize the product distribution metrics by various tools and services, such as Giovanni, the Open-source Project for a Network Data Access Protocol (OPeNDAP), and subsets, to address how these tools/services have extended the usage of data in the EOSDIS collection. Data usage patterns based on discipline and study area will further assist in understanding how EOSDIS data user needs have evolved over time. Results from this study will provide useful information for the DAACs that can help them improve the functionality of their tools and services as well as more efficiently allocate the resources necessary for enhanced access and availability of their data products. Knowledge of these metrics may also benefit user discovery of data in the EOSDIS collection, promote research collaboration, and stimulate new ideas from work and research conducted using specific datasets and data collections.
Kafle, Durga N.↗