Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “science data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 541 records · Page 30
All sourcesNASA NTRSDOE OSTI
NASA NTRS2006

A Multi-mission Event-Driven Component-Based System for Support of Flight Software Development, ATLO, and Operations first used by the Mars Science Laboratory (MSL) Project

This viewgraph presentation reviews the architectural description of the Mission Data Processing and Control System (MPCS). MPCS is an event-driven, multi-mission ground data processing components providing uplink, downlink, and data management capabilities which will support the Mars Science Laboratory (MSL) project as its first target mission. MPCS is designed with these factors (1) Enabling plug and play architecture (2) MPCS has strong inheritance from GDS components that have been developed for other Flight Projects (MER, MRO, DAWN, MSAP), and are currently being used in operations and ATLO, and (3) MPCS components are Java-based, platform independent, and are designed to consume and produce XML-formatted data

Mission Data Processing and Control System (MPCS)↗
NASA NTRS2019

Developing a Machine-Learning-Based Processing Framework for Twitter and Other Crowdsourced Data

Crowdsourced data streams such as Twitter and other social media are important sources of real-time and historical global information for Earth science applications. At the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), we have been exploring the Twitter data stream for its potential in augmenting the validation program of NASA's Global Precipitation Measurement (GPM) mission. To realize this potential, we need to increase the information density and enhance the quality of filtered precipitation tweets. We have implemented various components of a machine learning (ML)-based processing infrastructure for crowdsourced data that outputs, in this instance, useful and usable information derived from precipitation tweets. We have test enriched the Twitter stream with higher quality active tweets from those knowingly contributing to our effort and from existing crowdsourced programs (e.g., mPING, CoCoRaHS). We have experimented with various algorithms for processing tweets, including Naà ve Bayes, Convolutional Neural Network (CNN), Hierarchical Attention Network (HAN), and semi-supervised learning (with tri-training). Our current work focuses on (1) automated review of Earth science-related publications to determine relationships between discipline research needs and ML algorithms; (2) investigating Sequential Generative Adversarial Network (SeqGAN) for processing precipitation tweets for anomaly detection; and (3) managing crowdsourced data in a way that is compatible with existing NASA satellite data archives and using the data for ML applications. Key results include (1) network visualization of NLP-processed publications in various Earth science disciplines; (2) difference between GPM-linked, generated tweets and collected actual tweets that is small for GPM-determined light to moderate rain cases and high for GPM-determined heavy rain cases; and (3) identification of MongoDB for storing raw tweets and Zarr format for gridded tweets (compatible with GPM data). Our results have taken us a step closer to an operational ML-based tweet processing infrastructure and have already demonstrated that tweet-derived precipitation information is potentially useful for validation of Earth science satellite data.

Teng, William↗
DOE OSTI2017

Python in the NERSC Exascale Science Applications Program for Data

We describe a new effort at the National Energy Re- search Scientific Computing Center (NERSC) in performance analysis and optimization of scientific Python applications targeting the Intel Xeon Phi (Knights Landing, KNL) many- core architecture. The Python-centered work outlined here is part of a larger effort called the NERSC Exascale Science Applications Program (NESAP) for Data. NESAP for Data focuses on applications that process and analyze high-volume, high-velocity data sets from experimental/observational science (EOS) facilities supported by the US Department of Energy Office of Science. We present three case study applications from NESAP for Data that use Python. These codes vary in terms of “Python purity” from applications developed in pure Python to ones that use Python mainly as a convenience layer for scientists without expertise in lower level programming lan- guages like C, C++ or Fortran. The science case, requirements, constraints, algorithms, and initial performance optimizations for each code are discussed. Our goal with this paper is to contribute to the larger conversation around the role of Python in high-performance computing today and tomorrow, highlighting areas for future work and emerging best practices

Ronaghi, Zahra↗
NASA NTRSDate not supplied

Recent Progress at the International Laser Ranging Service (ILRS)

The International Laser Ranging Service (ILRS) is experiencing significant growth. New stations are being built and brought into operation, expanding the spatial and temporal coverage of the Satellite Laser Ranging (SLR) observations. Some of these stations are multi-technique Core Sites that allow us to exploit the combined strengths of different geodetic space techniques. New satellites are strengthening the ILRS contribution to the International Terrestrial Reference Frame (ITRF), and expanding the spectrum of satellite applications supported by the ILRS. New analysis, modeling, and data processing techniques continue to improve the science products. New campaigns are continuing to expand SLR applications into relativity and the study of non-gravitational forces. The ILRS is currently participating in the Galileo for Science campaign (G4S_2.0 project), funded by the Italian Space Agency (ASI), which aims to perform a set of measurements in the field of gravitation with the Galileo satellites, taking advantage of the accuracy of the on-board atomic clocks and, in particular, of GSAT0201and GSAT0202 by exploiting their relatively high eccentricity (~0.16). New activities underway in Lunar Laser Ranging (LLR), with the imminent launch and deployment of the upcoming Next Generation Lunar Retroreflector - 1 (NGLR-1) to the lunar surface. Some new satellites supported by the ILRS are now exploring means of capturing Space Debris, while some ILRS stations run local laser ranging projects to track space debris, contributing to the maintenance of various data catalogs. Time transfer experiments are also expanding the application of laser ranging, linking two fully characterized optical clocks at different locations with SLR (Einstein Synchronization) providing measurements of height differences, demonstrating the feasibility of a physical height system. Two-way and one-way transponders activity has been underway since the LASSO experiment in the 1990’s. More experiments are underway or planned. Applications include time and frequency transfer and clock synchronization between ground and space, interplanetary laser ranging, and fundamental physics. This presentation will give an update on ILRS activities and the impact that these activities have had and will have on ILRS data products.

Lunar Laser Ranging↗
DOE OSTI2018

2K-Capture in {sup 124}Хе: Results of Data Processing for an Exposure of 37.7 kg day

The results of the experimental search for two-neutrino 2K-capture in {sup 124}Xe with a large copper proportional counter obtained by processing the data for an exposure of 37.7 kg day are presented. The experimental setup is located at the Underground Low-Background: Laboratory of the Baksan Neutrino Observatory at a depth of 4900 m.w.e. The combination of methods of selection of useful signals with a unique set of characteristics and the event topology taken into account allowed us to suppress the background in the energy region of interest. A new half-life limit for 2K(2ν)-capture in {sup 124}Xe was determined: T{sub 1/2} ≥ 7.7 × 10{sup 21} years (90% CL).

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NASA NTRSDate not supplied

Satellite Instrument Calibration

Satellite remote sensing is an integral part of global earth observation. Satellite remote sensing data enable us to observe the whole globe at a reasonably high frequency. To maintain the accuracy of satellite remote sensing data, it is critical to continuously perform instrument calibration throughout the lifetime of the mission. This presentation describes the common characteristics of earth observing satellite sensors, the calibration processes and potential issues, the implication on science data, and how we understand the globe.

calibration↗
DOE OSTI2026

CMIP7 Data Request: atmosphere priorities and opportunities

This paper presents a comprehensive overview of the Coupled Model Intercomparison Project Phase 7 (CMIP7) request for data unlocking key research avenues in atmospheric science and provides justification for the resources needed to produce this data. Topics within the CMIP7 Atmosphere Theme centre around processes and feedbacks in atmospheric science such as clouds, aerosols and atmospheric chemistry, atmospheric circulation, temperature variability and extremes, radiative forcings, and Earth system model evaluation. These topics are summarised in this paper as scientific “opportunities” which will be realised through CMIP7 experiments and Earth system model outputs. These opportunities were submitted by a thematic group of atmospheric science community representatives combined with an extended consultation process. The production of these variables will close key gaps and uncertainties identified during previous rounds of CMIP, and will be broadly used by scientific, policy, governmental, industry, and other communities that rely on climate model projections for research and decision making, including supporting the 7th Intergovernmental Panel on Climate Change Assessment Report (AR7). As an author group, we also reflect on the process used to collate this data request and make recommendations to future CMIP governance on implementing a consultation on this scale in the future.

58 GEOSCIENCES↗
NASA NTRS2012

Communicating by Doppler: Detecting Spacecraft Dynamics During Critical Maneuvers

Communicating information from spacecraft in deep space utilizes sophisticated techniques of modulations onto a microwave signal carrier. Under most conditions, there is a high success rate of sending commands to the spacecraft and receiving science data acquired by on-board instruments along with health and status information. There are conditions, however, where the signal dynamics are too high and/or the received signal-to-noise ratio is below the receiver threshold. Under these conditions, often by design and sometimes as a result of planned or unplanned critical maneuvers, events (e.g., orbit insertion or descent and landing), safe mode, etc., it becomes highly critical but exceedingly challenging to receive information about the health and dynamical behavior of the spacecraft. The Deep Space Network, being a world-class instrument for Radio Science research, developed openloop receivers, called the Radio Science Receiver, designed to capture the raw incoming electromagnetic signals and associated noise for Radio Science experiments; post data capture digital signal processing extracts the signal carrier for scientific analysis. This receiver provides a high level of configuration flexibility and can be optimized for the various types of experiments. In addition to its scientific utility, it proved to be useful, and in some cases critical, for the support of missions during specific scenarios were the link budget is below the threshold of the tracking receiver to maintain lock or the frequency dynamics are faster than the limits of the tracking receiver. In these cases, the signal carrier is often detected only in the open-loop receiver to provide information on the specific behavior of the spacecraft from the carrier dynamics. This paper describes the utility of the system to support mission-critical events for the three cases of Cassini's Saturn orbit insertion, Huygens Titan landing, and Mars rovers landing.

radio science↗
NASA NTRS1994

Multimission Telemetry Visualization (MTV) system: A mission applications project from JPL's Multimedia Communications Laboratory

This paper describes the Multimission Telemetry Visualization (MTV) data acquisition/distribution system. MTV was developed by JPL's Multimedia Communications Laboratory (MCL) and designed to process and display digital, real-time, science and engineering data from JPL's Mission Control Center. The MTV system can be accessed using UNIX workstations and PC's over common datacom and telecom networks from worldwide locations. It is designed to lower data distribution costs while increasing data analysis functionality by integrating low-cost, off-the-shelf desktop hardware and software. MTV is expected to significantly lower the cost of real-time data display, processing, distribution, and allow for greater spacecraft safety and mission data access.

Koeberlein, Ernest, III↗
NASA NTRS2016

Comparative Results of AIRS AMSU and CrIS/ATMS Retrievals Using a Scientifically Equivalent Retrieval Algorithm

The AIRS Science Team Version 6 retrieval algorithm is currently producing high quality level-3 Climate Data Records (CDRs) from AIRSAMSU which are critical for understanding climate processes. The AIRS Science Team is finalizing an improved Version-7 retrieval algorithm to reprocess all old and future AIRS data. AIRS CDRs should eventually cover the period September 2002 through at least 2020. CrISATMS is the only scheduled follow on to AIRSAMSU. The objective of this research is to prepare for generation of a long term CrISATMS level-3 data using a finalized retrieval algorithm that is scientifically equivalent to AIRSAMSU Version-7.

AIRS↗
NASA NTRS2016

Comparative Results of AIRS/AMSU and CrIS/ATMS Retrievals Using a Scientifically Equivalent Retrieval Algorithm

The AIRS Science Team Version-6 retrieval algorithm is currently producing high quality level-3 Climate Data Records (CDRs) from AIRS/AMSU which are critical for understanding climate processes. The AIRS Science Team is finalizing an improved Version-7 retrieval algorithm to reprocess all old and future AIRS data. AIRS CDRs should eventually cover the period September 2002 through at least 2020. CrIS/ATMS is the only scheduled follow on to AIRS/AMSU. The objective of this research is to prepare for generation of long term CrIS/ATMS CDRs using a retrieval algorithm that is scientifically equivalent to AIRS/AMSU Version-7.

AIRS AMSU↗
NASA NTRS2011

Generic, Extensible, Configurable Push-Pull Framework for Large-Scale Science Missions

The push-pull framework was developed in hopes that an infrastructure would be created that could literally connect to any given remote site, and (given a set of restrictions) download files from that remote site based on those restrictions. The Cataloging and Archiving Service (CAS) has recently been re-architected and re-factored in its canonical services, including file management, workflow management, and resource management. Additionally, a generic CAS Crawling Framework was built based on motivation from Apache s open-source search engine project called Nutch. Nutch is an Apache effort to provide search engine services (akin to Google), including crawling, parsing, content analysis, and indexing. It has produced several stable software releases, and is currently used in production services at companies such as Yahoo, and at NASA's Planetary Data System. The CAS Crawling Framework supports many of the Nutch Crawler's generic services, including metadata extraction, crawling, and ingestion. However, one service that was not ported over from Nutch is a generic protocol layer service that allows the Nutch crawler to obtain content using protocol plug-ins that download content using implementations of remote protocols, such as HTTP, FTP, WinNT file system, HTTPS, etc. Such a generic protocol layer would greatly aid in the CAS Crawling Framework, as the layer would allow the framework to generically obtain content (i.e., data products) from remote sites using protocols such as FTP and others. Augmented with this capability, the Orbiting Carbon Observatory (OCO) and NPP (NPOESS Preparatory Project) Sounder PEATE (Product Evaluation and Analysis Tools Elements) would be provided with an infrastructure to support generic FTP-based pull access to remote data products, obviating the need for any specialized software outside of the context of their existing process control systems. This extensible configurable framework was created in Java, and allows the use of different underlying communication middleware (at present, both XMLRPC, and RMI). In addition, the framework is entirely suitable in a multi-mission environment and is supporting both NPP Sounder PEATE and the OCO Mission. Both systems involve tasks such as high-throughput job processing, terabyte-scale data management, and science computing facilities. NPP Sounder PEATE is already using the push-pull framework to accept hundreds of gigabytes of IASI (infrared atmospheric sounding interferometer) data, and is in preparation to accept CRIMS (Cross-track Infrared Microwave Sounding Suite) data. OCO will leverage the framework to download MODIS, CloudSat, and other ancillary data products for use in the high-performance Level 2 Science Algorithm. The National Cancer Institute is also evaluating the framework for use in sharing and disseminating cancer research data through its Early Detection Research Network (EDRN).

Foster, Brian M.↗
DOE OSTI2021

Development of Explainable, Knowledge-Guided AI Models to Enhance the E3SM Land Model Development and Uncertainty Quantification

Focal Area(s): (2)Predictive modeling using AI techniques and AI-derived model components; use of AI and other tools to design a prediction system comprising of a hierarchy of models. (3) Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge- guided AI. Science Challenge: The Energy Exascale Earth System Model (E3SM) is a fully coupled, state-of-the-science Earth system model that uses code optimized for DOE's advanced computers to address the most critical scientific questions facing our nation and society (Golaz et al., 2019). The E3SM Land model (ELM) is designed to understand how the changes in terrestrial land surfaces will interact with other Earth system components and has been used to understand hydrologic cycles, biogeophysics, and ecosystem dynamics. In spite of great successes, the ELM has several known issues that restrain rapid improvements. For example, the ELM uses equilibrium models to simulate dynamic land-climate interactions and it requires long model spin-up time to identify suitable initial conditions for transient simulations. The ELM lacks built-in uncertainty mechanisms that can improve the robustness of model predictions. The ELM is a holistic, deterministic model system with a rigid design, and in many situations, it is hard to modify the ELM system to incorporate new theory/hypothesis and new data across scales to address emerging science problems (such as predicting the impacts of water cycle extremes). In addition, The ELM is technically optimized for traditional CPU-centric computers and it cannot fully utilize the current and incoming leadership computers for model simulations and uncertainty quantification (UQ). The success of artificial intelligence (AI) has inspired scientists to use AI models to discover intrinsic features from simulation data (Chattopadhyay et al., 2020) and observational data (Reichstein et al., 2019) to gain further process understanding of Earth science problems. However, autonomous AI model training through deep learning usually requires a huge amount of annotated data. To overcome the limitations from the data and computing resources, knowledge-guided AI models are necessary where human-knowledge is ingested in model construction (Banino et al., 2018) and training process (Silver et al., 2016) for efficient learning. Herein, we present a new way that leverages the process understanding from the ELM to guide AI model development for the ELM enhancement and UQ. We hope this study can inspire further Earth and environmental system model developments and transformations.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2014

GPM Data Products, Their Availability, and Production Status

On February 28, 2014, Japan Standard Time, the Global Precipitation Measurement (GPM) mission was launched in a picture-perfect launch activity. On March 4, 2014, the GPM Microwave Imager (GMI) was put into science observation mode. The Dual-Frequency Radar (DPR) was put in science observation mode on March 8, 2014. The Precipitation Processing System (PPS) produced products immediately upon receiving the data. Both regular science products and Near-realtime (NRT) products were produced. These were made immediately available to a group of early adopters. In mid-June 2014, GMI level-1 brightness temperature products were made publicly available. In mid-July 2014, GMI and partner-radiometer precipitation retrievals were made public. GMI public availability was several months ahead of the planned release. The DPR products became publicly available on the planned release date of September 2, 2014. Data continues to be available to any user desiring it.

Data Products↗
NASA NTRS2010

Pixel-Level Calibration in the Kepler Science Operations Center Pipeline

We present an overview of the pixel-level calibration of flight data from the Kepler Mission performed within the Kepler Science Operations Center Science Processing Pipeline. This article describes the calibration (CAL) module, which operates on original spacecraft data to remove instrument effects and other artifacts that pollute the data. Traditional CCD data reduction is performed (removal of instrument/detector effects such as bias and dark current), in addition to pixel-level calibration (correcting for cosmic rays and variations in pixel sensitivity), Kepler-specific corrections (removing smear signals which result from the lack of a shutter on the photometer and correcting for distortions induced by the readout electronics), and additional operations that are needed due to the complexity and large volume of flight data. CAL operates on long (~30 min) and short (~1 min) sampled data, as well as full-frame images, and produces calibrated pixel flux time series, uncertainties, and other metrics that are used in subsequent Pipeline modules. The raw and calibrated data are also archived in the Multi-mission Archive at Space Telescope at the Space Telescope Science Institute for use by the astronomical community.

Calibration↗
NASA NTRSDate not supplied

Updates in Developing a Prototype Science Pipeline and Full-Volume, Global Hyperspectral Synthetic Data Sets for NASA’s Earth System Observatory’s Upcoming Surface, Biology and Geology Mission

The Surface Biology and Geology (SBG) mission recently passed mission confirmation review and has entered phase A – design and development. SBG will acquire high resolution solar-reflected spectroscopy and thermal infrared observations at a data rate of ~2.5 TB/day and generate products at ~40 TB/day. Given that the per-day volume is greater than NASA’s total extant airborne hyperspectral data collection, collecting, processing, disseminating, and exploiting the SBG data present new challenges. To meet these challenges, we have developed a prototype science pipeline and a full-volume global hyperspectral synthetic data set to help prepare for SBG’s flight (see poster GC42D-0730). Our science pipeline is based on the science processing technology developed for NASA’s Kepler and TESS planet-hunting missions. The pipeline infrastructure, Ziggy, provides a scalable architecture for robust, repeatable, and replicable science and application products that can be run on a range of systems from a laptop to the cloud or a supercomputer. Ziggy is compliant with NASA Procedural Requirement (NPR) 7150.2C, is at a technical readiness level (TRL) of 7 and has been released to github.com/nasa/ziggy. We integrated Ziggy with EO-1/Hyperion workflows to build a prototype pipeline and ingested the 17-year mission archive that provides globally sampled visible through shortwave infrared spectra that are representative of SBG data types and volumes. We fully implemented the first stage and processed the entire 55 TB Hyperion data set from the raw data (Level 0) to top-of-the-atmosphere radiance (Level 1R). We are currently evaluating the ISOFIT atmospheric correction module to convert the L1R data to surface reflectance (Level 2) before reprocessing the full data set to L2. Crosschecks are being performed with RadCalNet as well as with coincident observations by AVIRIS. We are also investigating modern methods for georectifying the Hyperion scenes. Finally, we describe an analysis of the cost to conduct forward processing and reprocessing campaigns for SBG on HECC with dedicated compute and storage resources using the resurrected Hyperion pipeline as a proxy for full-volume SBG data. The analysis demonstrates that SBG L0 data can be processed to L2 on HECC with full reprocessing campaigns every two years for ~$2.6M over a 7-year lifespan. Moreover, 69% of the system capacity would be available for other activities, possibly enabling future open-source science activities, including algorithm development, L3+ processing, .etc.

ESD↗
NASA NTRS2003

Automated Data Processing as an AI Planning Problem

NASA s vision for Earth Science is to build a "sensor web"; an adaptive array of heterogeneous satellites and other sensors that will track important events, such as storms, and provide real-time information about the state of the Earth to a wide variety of customers. Achieving his vision will require automation not only in the scheduling of the observations but also in the processing af tee resulting data. Ta address this need, we have developed a planner-based agent to automatically generate and execute data-flow programs to produce the requested data products. Data processing domains are substantially different from other planning domains that have been explored, and this has led us to substantially different choices in terms of representation and algorithms. We discuss some of these differences and discuss the approach we have adopted.

Golden, Keith↗
NASA NTRS2020

Famine Early Warning Systems Network (FEWS NET) Land Data Assimilation System (LDAS) and Other Assimilated Hydrological Data at NASA GES DISC

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) provides science support for several data sets relevant to agriculture and food security, including the Famine Early Warning Systems Network (FEWS NET) Land Data Assimilation System (LDAS), or FLDAS data set. The GES DISC is one of twelve NASA Earth Observing System (EOS) data centers that process, archive, document, and distribute data from Earth science missions and related projects. The GES DISC hosts a wide range of remote sensing and model data, and provides reliable and robust data access and other services to users worldwide. Beyond data archive and access, the GES DISC offers many services to visualize and analyze the data. This presentation provides a summary of the hydrological data available at the GES DISC, along with an overview of related data services. Specifically, the FLDAS data set has been adapted to work with domains, data streams, and monitoring and forecast requirements associated with food security assessment in data-sparse, developing country settings. The FLDAS global monthly data have a 0.1 x 0.1 degree spatial resolution covering the period from January 1982 to present. Global FLDAS monthly anomaly and monthly climatology data are also available at the GES DISC to evaluate how current conditions compare to averages over the FLDAS 35-year period. Several case studies using the FLDAS soil moisture, evapotranspiration, rainfall, runoff, and surface temperature data will be presented.

Loeser, Carlee↗