Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “production data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Shallow cumuli cover and its uncertainties from ground-based lidar–radar data and sky images

Cloud cover estimates of single-layer shallow cumuli obtained from narrow field-of-view (FOV) lidar–radar and wide-FOV total sky imager (TSI) data are compared over an extended period (2000–2017 summers) at the established United States Atmospheric Radiation Measurement mid-continental Southern Great Plains site. We quantify the impacts of two factors on hourly and sub-hourly cloud cover estimates: (1) instrument-dependent cloud detection and data merging criteria and (2) FOV configuration. Enhanced observations at this site combine the advantages of the ceilometer, micropulse lidar (MPL) and cloud radar in merged data products. Data collected by these three instruments are used to calculate narrow-FOV cloud fraction (CF) as a temporal fraction of cloudy returns within a given period. Sky images provided by TSI are used to calculate the wide-FOV fractional sky cover (FSC) as a fraction of cloudy pixels within a given image. To assess the impact of the first factor on CF obtained from the merged data products, we consider two additional subperiods (2000–2010 and 2011–2017 summers) that mark significant instrumentation and algorithmic advances in the cloud detection and data merging. We demonstrate that CF obtained from ceilometer data alone and FSC obtained from sky images provide the most similar and consistent cloud cover estimates; hourly bias and root-mean-square difference (RMSD) are within 0.04 and 0.12, respectively. However, CF from merged MPL–ceilometer data provides the largest estimates of the multiyear mean cloud cover, about 0.12 (35 %) and 0.08 (24 %) greater than FSC for the first and second subperiods, respectively. CF from merged ceilometer–MPL–radar data has the strongest subperiod dependence with a bias of 0.08 (24 %) compared to FSC for the first subperiod and shows no bias for the second subperiod. The strong period dependence of CF obtained from the combined ceilometer–MPL–radar data is likely results from a change in what sensors are relied on to detect clouds below 3 km. After 2011, the MPL stopped being used for cloud top height detection below 3 km, leaving the radar as the only sensor used in cloud top height retrievals. To quantify the FOV impact, a narrow-FOV FSC is derived from the TSI images. We demonstrate that FOV configuration does not modify the bias but impacts the RMSD (0.1 hourly, 0.15 sub-hourly). In particular, the FOV impact is significant for sub-hourly observations, where 41 % of narrow- and wide-FOV FSC differ by more than 0.1. A new “quick-look” tool is introduced to visualize impacts of these two factors through integration of CF and FSC data with novel TSI-based images of the spatial variability in cloud cover. The influence of cloud field organization, such cloud streets parallel to the wind direction, on narrow- and wide-FOV cloud cover estimates can be visually assessed.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2004

Enhancing the IGS Data and Products Infrastructure: A Data Center Perspective

The Data Transfer and Data Centers session will present papers on current data center activities as well as various topics geared towards enhancing the availability and acquisition of International Global Positions System Service (IGS) data and products. These topics include: strategies for reducing data file latency at IGS Data Centers, standardization of data file replacement notification methods, improving the efficiency of data file transfer among IGS Data Centers, and defining a network topology for GPS data files for use in applications.

Scharber, Michael↗
DOE OSTI2025

Raw ERCOT 60-Day SCED Disclosure Reports

This dataset contains ERCOT plant-level wind power production data at a 15-minute time resolution. Data at ERCOT are called 60-Day SCED Disclosure Report; data product number NP3-965-ER. Raw data are in local time. DST is handled as "skip hour" (12a, 1a, 3a,...) in spring, "extra hour" (12a, 1a, 2a, 2a, 3a,...) in fall.

17 WIND ENERGY↗
DOE OSTI2022

Global Carbon Budget 2021

Abstract. Accurate assessment of anthropogenic carbon dioxide (CO2) emissions and their redistribution among the atmosphere, ocean, and terrestrial biosphere in a changing climate is critical to better understand the global carbon cycle, support the development of climate policies, and project future climate change. Here we describe and synthesize datasets and methodology to quantify the five major components of the global carbon budget and their uncertainties. Fossil CO2 emissions (EFOS) are based on energy statistics and cement production data, while emissions from land-use change (ELUC), mainly deforestation, are based on land use and land-use change data and bookkeeping models. Atmospheric CO2 concentration is measured directly, and its growth rate (GATM) is computed from the annual changes in concentration. The ocean CO2 sink (SOCEAN) is estimated with global ocean biogeochemistry models and observation-based data products. The terrestrial CO2 sink (SLAND) is estimated with dynamic global vegetation models. The resulting carbon budget imbalance (BIM), the difference between the estimated total emissions and the estimated changes in the atmosphere, ocean, and terrestrial biosphere, is a measure of imperfect data and understanding of the contemporary carbon cycle. All uncertainties are reported as ±1σ. For the first time, an approach is shown to reconcile the difference in our ELUC estimate with the one from national greenhouse gas inventories, supporting the assessment of collective countries' climate progress. For the year 2020, EFOS declined by 5.4 % relative to 2019, with fossil emissions at 9.5 ± 0.5 GtC yr−1 (9.3 ± 0.5 GtC yr−1 when the cement carbonation sink is included), and ELUC was 0.9 ± 0.7 GtC yr−1, for a total anthropogenic CO2 emission of 10.2 ± 0.8 GtC yr−1 (37.4 ± 2.9 GtCO2). Also, for 2020, GATM was 5.0 ± 0.2 GtC yr−1 (2.4 ± 0.1 ppm yr−1), SOCEAN was 3.0 ± 0.4 GtC yr−1, and SLAND was 2.9 ± 1 GtC yr−1, with a BIM of −0.8 GtC yr−1. The global atmospheric CO2 concentration averaged over 2020 reached 412.45 ± 0.1 ppm. Preliminary data for 2021 suggest a rebound in EFOS relative to 2020 of +4.8 % (4.2 % to 5.4 %) globally. Overall, the mean and trend in the components of the global carbon budget are consistently estimated over the period 1959–2020, but discrepancies of up to 1 GtC yr−1 persist for the representation of annual to semi-decadal variability in CO2 fluxes. Comparison of estimates from multiple approaches and observations shows (1) a persistent large uncertainty in the estimate of land-use changes emissions, (2) a low agreement between the different methods on the magnitude of the land CO2 flux in the northern extra-tropics, and (3) a discrepancy between the different methods on the strength of the ocean sink over the last decade. This living data update documents changes in the methods and datasets used in this new global carbon budget and the progress in understanding of the global carbon cycle compared with previous publications of this dataset (Friedlingstein et al., 2020, 2019; Le Quéré et al., 2018b, a, 2016, 2015b, a, 2014, 2013). The data presented in this work are available at https://doi.org/10.18160/gcp-2021 (Friedlingstein et al., 2021).

Friedlingstein, Pierre (ORCID:0000000333094739)↗
NASA NTRS2005

Calibration Refinements in Support of MISR

The Multi-angle Imaging SpectroRadiometer (MISR) is one of five instruments on-board the EOSI Terra spacecraft, and has collected science data since March 2000. A multi-angle capability is provided by ninecameras, which view up to 70' forward and aft of the spacecraft track and enable unique geophysical retrievals. Throughout its mission, a calibration team has made periodic refinements to the process used to calibrate MISR. These have resulted in improved absolute, and band and camera-relative calibrations, as well as in derived geophysical data products. Data reprocessing is on-going, such that these refinements also improve previous data acquisitions. The calibration process is believed to be mature at this time, with no other changes anticipated. Bimonthly deployments of the on-board-calibrator continue to monitor instrument response degradations, and provide correction coefficients needed to maintain the accuracy of the radiance products.

refinements↗
NASA NTRS1978

Landsat missions

This paper gives a description and present status of NASA's Landsat program, including the orbital coverage, payload, data processing, and data products. Data uses are discussed in general terms. Also discussed are NASA's plans for the Landsat-3 and Landsat-D missions and the improved capabilities represented by these missions. The Shuttle retrieval of Landsat-D and the Shuttle launch of future Landsat satellites are discussed conceptually.

Mannheimer, H.↗
NASA NTRS1989

The moderate resolution imaging spectrometer data system

Science requirements for a moderate resolution imaging spectrometer (MODIS) data system are outlined, as well as the operations concept and system specifications based on these requirements. Two systems are considered - MODIS-N for a 40-channel nadir-viewing scanner and MODIS-T for a 64-channel tiltable scanner. The first system collects data from 15 thermal-infrared channels at all times and from 25 reflected-energy channels during daytime, while MODIS-T will take data from 64 channels on a 100-percent duty cycle during daytime only. Standard data products, data acquisition and processing, and requirements on data are discussed. Emphasis is placed on the interfaces between the science data processing facilities and the scientific users.

Han, Daesoo↗
NASA NTRS2001

A Metadata Action Language

The data management problem comprises data processing and data tracking. Data processing is the creation of new data based on existing data sources. Data tracking consists of storing metadata descriptions of available data. This paper addresses the data management problem by casting it as an AI planning problem. Actions are data-processing commands, plans are dataflow programs and goals are metadata descriptions of desired data products. Data manipulation is simply plan generation and execution, and a key component of data tracking is inferring the effects of an observed plan. We introduce a new action language for data management domains, called ADILM. We discuss the connection between data processing and information integration and show how a language for the latter must be modified to support the former. The paper also discusses information gathering within a data-processing framework, and show how ADILM metadata expressions are a generalization of Local Completeness.

Golden, Keith↗
NASA NTRS2003

Automated Data Processing as an AI Planning Problem

NASA s vision for Earth Science is to build a "sensor web"; an adaptive array of heterogeneous satellites and other sensors that will track important events, such as storms, and provide real-time information about the state of the Earth to a wide variety of customers. Achieving his vision will require automation not only in the scheduling of the observations but also in the processing af tee resulting data. Ta address this need, we have developed a planner-based agent to automatically generate and execute data-flow programs to produce the requested data products. Data processing domains are substantially different from other planning domains that have been explored, and this has led us to substantially different choices in terms of representation and algorithms. We discuss some of these differences and discuss the approach we have adopted.

Golden, Keith↗
NASA NTRS2016

Addressing the Big-Earth-Data Variety Challenge with the Hierarchical Triangular Mesh

We have implemented an updated Hierarchical Triangular Mesh (HTM) as the basis for a unified data model and an indexing scheme for geoscience data to address the variety challenge of Big Earth Data. We observe that, in the absence of variety, the volume challenge of Big Data is relatively easily addressable with parallel processing. The more important challenge in achieving optimal value with a Big Data solution for Earth Science (ES) data analysis, however, is being able to achieve good scalability with variety. With HTM unifying at least the three popular data models, i.e. Grid, Swath, and Point, used by current ES data products, data preparation time for integrative analysis of diverse datasets can be drastically reduced and better variety scaling can be achieved. In addition, since HTM is also an indexing scheme, when it is used to index all ES datasets, data placement alignment (or co-location) on the shared nothing architecture, which most Big Data systems are based on, is guaranteed and better performance is ensured. Moreover, our updated HTM encoding turns most geospatial set operations into integer interval operations, gaining further performance advantages.

SciDB↗
NASA NTRS2012

Ocean Surface Vector Wind: Research Challenges and Operational Opportunities

The atmosphere and ocean are joined together over seventy percent of Earth, with ocean surface vector wind (OSVW) stress one of the linkages. Satellite OSVW measurements provide estimates of wind divergence at the bottom of the atmosphere and wind stress curl at the top of the ocean; both variables are critical for weather and climate applications. As is common with satellite measurements, a multitude of OSVW data products exist for each currently operating satellite instrument. In 2012 the Joint Technical Commission on Oceanography and Marine Meteorology (JCOMM) launched an initiative to coordinate production of OSVW data products to maximize the impact and benefit of existing and future OSVW measurements in atmospheric and oceanic applications. This paper describes meteorological and oceanographic requirements for OSVW data products; provides an inventory of unique data products to illustrate that the challenge is not the production of individual data products, but the generation of harmonized datasets for analysis and synthesis of the ensemble of data products; and outlines a vision for JCOMM, in partnership with other international groups, to assemble an international network to share ideas, data, tools, strategies, and deliverables to improve utilization of satellite OSVW data products for research and operational applications.

Joint Technical Commission on Oceanography and Mar↗
DOE OSTI2026

Vera C. Rubin Observatory Prompt Products: alert packets data

Data products produced by prompt and daily processing of images obtained in the Legacy Survey of Space and Time. These include realtime alerts sent to community alert brokers, newly-discovered Solar System Objects reported to the Minor Planet Center, processed visit and difference images, and source catalogs. Prompt Products are not a static single data release but continually grow throughout the ten-year LSST survey. This dataset is a subset of the full data release consisting of a dataset named alert packets. This dataset contains measurements for 5-sigma sources detected in difference images that were issued to the community brokers.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2004

The Development of Two Science Investigator-led Processing Systems (SIPS) for NASA's Earth Observation System (EOS)

In 2001, NASA Goddard Space Flight Center's Laboratory for Terrestrial Physics started the construction of a science Investigator-led Processing System (SIPS) for processing data from the Ozone Monitoring Instrument (OMI) which will launch on the Aura platform in mid 2004. The Ozone Monitoring Instrument (OMI) is a contribution of the Netherlands Agency for Aerospace Programs (NIVR) in collaboration with the Finnish Meteorological Institute (FMI) to the Earth Observing System (EOS) Aura mission. It will continue the Total Ozone Monitoring System (TOMS) record for total ozone and other atmospheric parameters related to ozone chemistry and climate. OMI measurements will be highly synergistic with the other instruments on the EOS Aura platform. The LTP previously developed the Moderate Resolution Imaging Spectrometer (MODIS) Data Processing System (MODAPS), which has been in full operations since the launches of the Terra and Aqua spacecrafts in December, 1999 and May, 2002 respectively. During that time, it has continually evolved to better support the needs of the MODIS team. We now run multiple instances of the system managing faster than real time reprocessings of the data as well as continuing forward processing. The new OMI Data Processing System (OMIDAPS) was adapted from the MODAPS. It will ingest raw data from the satellite ground station and process it to produce calibrated, geolocated higher level data products. These data products will be transmitted to the Goddard Distributed Active Archive Center (GDAAC) instance of the Earth Observing System (EOS) Data and Information System (EOSDIS) for long term archive and distribution to the public. The OMIDAPS will also provide data distribution to the OMI Science Team for quality assessment, algorithm improvement, calibration, etc. We have taken advantage of lessons learned from the MODIS experience and software already developed for MODIS. We made some changes in the hardware system organization, database and software to adapt the system for OMI. We replaced the fundamental database system, Sybase, with an Open Source RDBMS called PostgreSQL, and based the entire OMIDAPS on a cluster of Linux based commodity computers rather than the large SGI servers that MODAPS uses. Rather than relying on a central I/O server host, the new system distributes its data archive among multiple server hosts in the cluster. OMI is also customizing the graphical user interfaces and reporting structure to more closely meet the needs of the OMI Science Team. Prior to 2003, simulated OMI data and the science algorithms were not ready for production testing. We initially constructed a prototype system and tested using a 25 year dataset of Total Ozone Mapping Spectrometer (TOMS) and Solar Backscatter Ultraviolet Instrument (SBUV) data. This prototype system provided a platform to support the adaptation of the algorithms for OMI, and provided reprocessing of the historical data aiding in its analysis. In a recent reanalysis of the TOMS data, the OMIDAPS processed 108,000 full orbits of data through 4 processing steps per orbit, producing about 800,000 files (400 GiB) of level 2 and greater data files. More recently we have installed two instances of the OMIDAPS for integration and testing of OM1 science processes as they get delivered from the Science Team. A Test instance of the OMIDAPS has also supported a series of "Interface Confidence Tests" (ICTs) and End-to-End Ground System tests to ensure the launch readiness of the system. This paper will discuss the high-level hardware, software, and database organization of the OMIDAPS and how it builds on the MODAPS heritage system. It will also provide an overview of the testing and implementation of the production OMIDAPS.

Tilmes, Curt↗
NASA NTRSDate not supplied

A User-Focused Renovation of CERES Metadata

Production software and public data products for Clouds and the Earth’s Radiant Energy System (CERES) continue to evolve as the project extends its climate data record. The data management team for CERES is currently undertaking major renovations of both code and data products, the latter of which is, of course, in service of improving user experience. A major mode of CERES’ data product improvement is in renovating products’ metadata. Metadata standards have evolved since CERES began producing its data products in 2000. In its twentieth year, CERES essentially asked the question: how would the project design its data products if it could start all over again? With forthcoming editions, this rebirth will be realized. CERES has redesigned its metadata standards to best position itself for data discoverability. The project has used the latest standards being developed in NASA’s Earth Science Data and Information Systems (ESDIS) Project’s Unified Metadata Model (UMM) documentation; collaborated with the Atmospheric Science Data Center (ASDC) to ensure compliance with Common Metadata Repository compatibility, and continued compliance with Climate and Forecast (CF) Conventions. In doing so, the team created its own, internal document for proper metadata creation and metadata verification software that is deployed prior to all code deliveries. This presentation will discuss this redesign process, as well as needs met and those that are still outstanding in the search for an improved user experience with CERES data products.

Kathleen Dejwakh↗
DOE OSTI2021

dbprocessing

dbprocessing is a python framework for automating data processing pipelines. The dbprocessing package uses a configuration file to define the input, intermediate, and output data products for a particular data type as well as the processes that connect those data products. In addition, dbprocessing uses python scripts called inspectors to determine if a particular file matches a configured data product. If a particular data product is found, it is ingested into the dbprocessing database (sqlite3 or postgres) which then triggers all of the chained processes to the final data product. While the framework is written in python, it can run software in any language but may require a “wrapper” to translate the dbprocessing command line arguments to the form expected by the software.

Walker, Andrew↗
DOE OSTI2025

Machine Learned Empirical Numerical Integrator from Simulated Data

Recently, a number of state-of-the-art surrogate machine learning (ML) models have been designed for global weather and climate prediction, which have been trained using reanalysis data products. Reanalysis data products are constructed using numerical model simulations that combine numerical integration of partial differential equations and parameterization schemes. These products are typically only archived and made available using coarsened spatial and temporal resolutions. This study explores the impact of the numerical generation methods used to produce the training datasets and the temporal resolution of those datasets on machine learning surrogate models. Using the nonlinear vector autoregression (NVAR) machine as an explainable ML technique, simple dynamical systems are emulated with ML models trained on data produced by three classical numerical integration schemes. NVAR is validated as a skillful ML method, capable of producing accurate predictions and, more importantly, reconstructing both the underlying dynamics and the numerical integration scheme used to generate the training data. However, the machine fails to generalize predictions on unseen test data generated by different numerical integration schemes, despite the underlying dynamical system being the same. This result provides a word of caution for the growing field of machine learning emulation of weather and climate dynamics. Furthermore, we illustrate using NVAR that training on temporally coarsened data may increase the required complexity of ML models and potentially introduce new numerical challenges. Finally, we discover that empirical integration schemes with arbitrary time-stepping sizes can be constructed directly from the data, which implies a potential for the development of empirical numerical integration schemes.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2007

Processing Raw HST Data With Up-to-Date Calibration Data

On-the-Fly Reprocessing (OTFR) is a collection of data-processing routines that work within the context of the Hubble Space Telescope (HST) pipeline data-flow system. The purpose served by OTFR is to generate, on demand, scientifically useful data products from raw HST data stored in an archive. First, on the basis of the requested final data products, OTFR retrieves the corresponding sets of raw data from the archives. Next, OTFR processes the raw data sets to remove artifacts and to establish proper header and other template information. Finally, the calibration routines appropriate to the specific data sets are invoked to produce the requested data products, and the data products are released to an archive distribution system for transmission to the requesting party. OTFR offers two notable advantages: (1) Inasmuch as calibrated data occupy about 8 times as much storage space as do raw data, by obviating storage of calibrated data, OTFR reduces the storage capacity needed by the archive; and (2) the calibration routines can be updated to give requesters the benefit of the most recent calibrations.

Miller, Warren↗
NASA NTRS2020

TESS Data Release Notes: Sector 24, DR35

These Data Release Notes provide information on the processing and export of data from the Transiting Exoplanet Survey Satellite (TESS). The data products included in this data release are full frame images (FFIs), target pixel fi les, light curve fi les, collateral pixel fi les, cotrending basis vectors (CBVs), and Data Validation (DV) reports, time series, and associated xml fi les. These data products were generated by the TESS Science Processing Operations Center (SPOC, Jenkins et al., 2016) at NASA Ames Research Center from data collected by the TESS instrument, which is managed by the TESS Payload Operations Center (POC) at Massachusetts Institute of Technology (MIT). The format and content of these data products are documented in the Science Data Products Description Document (SDPDD). The SPOC science algorithms are based heavily on those of the Kepler Mission science pipeline, and are described in the Kepler Data Processing Handbook (Jenkins, 2019).2 The Data Validation algorithms are documented in Twicken et al. (2018) and Li et al. (2019). The TESS Instrument Handbook (Vanderspek et al., 2018) contains more information about the TESS instrument design, detector layout, data properties, and mission operations. The TESS Mission is funded by NASA's Science Mission Directorate.

TESS↗