SEARCH · Engineering Papers

Results for “Science Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

All sources NASA NTRS DOE OSTI

NASA NTRS2001

A Geosynchronous Lidar System for Atmospheric Winds and Moisture Measurements

An observing system comprised of two lidars in geosychronous orbit would enable the synoptic and meso-scale measurement of atmospheric winds and moisture both of which are key first-order variables of the Earth's weather equation. Simultaneous measurement of these parameters at fast revisit rates promises large advancements in our weather prediction skills. Such capabilities would be unprecedented and a) yield greatly improved and finer resolution initial conditions for models, b) make existing costly and cumbersome measurement approaches obsolete, and c) obviate the use of numerical techniques needed to correct data obtained using present observing systems. Additionally, simultaneous synoptic wind and moisture observations would lead to improvements in model parameterizations, and in our knowledge of small-scale weather processes. Technology and science data product assessments are ongoing. Results will be presented during the conference.

Emmitt, G. D.↗

NASA NTRS2015

Architectures Toward Reusable Science Data Systems

Science Data Systems (SDS) comprise an important class of data processing systems that support product generation from remote sensors and in-situ observations. These systems enable research into new science data products, replication of experiments and verification of results. NASA has been building systems for satellite data processing since the first Earth observing satellites launched and is continuing development of systems to support NASA science research and NOAAs Earth observing satellite operations. The basic data processing workflows and scenarios continue to be valid for remote sensor observations research as well as for the complex multi-instrument operational satellite data systems being built today. System functions such as ingest, product generation and distribution need to be configured and performed in a consistent and repeatable way with an emphasis on scalability. This paper will examine the key architectural elements of several NASA satellite data processing systems currently in operation and under development that make them suitable for scaling and reuse. Examples of architectural elements that have become attractive include virtual machine environments, standard data product formats, metadata content and file naming, workflow and job management frameworks, data acquisition, search, and distribution protocols. By highlighting key elements and implementation experience we expect to find architectures that will outlast their original application and be readily adaptable for new applications. Concepts and principles are explored that lead to sound guidance for SDS developers and strategists.

Data Processing↗

NASA NTRS2002

TRMM Data Mining Service at the Goddard Earth Sciences (GES) DISC DAAC Tropical Rainfall Measuring Mission (TRMM)

TRMM has acquired more than four years of data since its launch in November 1997. All TRMM standard products are processed by the TRMM Science Data and Information System (TSDIS) and archived and distributed to general users by the GES DAAC. Table 1 shows the total archive and distribution as of February 28, 2002. The Utilization Ratio (UR), defined as the ratio of the number of distributed files to the number of archived files, of the TRMM standard products has been steadily increasing since 1998 and is currently at 6.98.

Source record↗

NASA NTRS2019

Standards and Best Practices - Two NASA Examples

Formal international standards as well as promotion of community or recommended practices have their place in ensuring "FAIRness" of data. Data management in NASA's Earth Observation System Data and Information System (EOSDIS) has benefited from both of these avenues to a significant extent. The purpose of this paper is to present one example of each of these, which promote (re)usability. The first is an ISO standard for specifying preservation content from Earth observation missions. The work on this started in 2011, informally within the Earth Science Information Partners (ESIP) in the US, while the European Space Agency (ESA) was leading an effort on Long-Term Data Preservation (LTDP). Resulting from the ESIP discussions was NASA's Preservation Content Specification, which was applied in 2012 as a requirement for NASA's new missions. ESA's Preserved Data Set Content (PDSC) document was codified into a document adopted by the Committee on Earth Observation Satellites (CEOS). It was recognized that it would be useful to combine PCS and PDSC into an ISO standard to ensure consistency in data preservation on a broader international scale. This standard, numbered ISO 19165-2 has been under development since mid-2017. The second is an example of developing recommendations for "best practices" within more limited (still fairly broad) communities. A Data Product Developers' Guide (DPDG) is currently being developed by one of NASA's Earth Science Data System Working Groups (ESDSWGs). It is for use by developers of products to be derived from Earth observation data to improve product (re)usability. One of the challenges in developing the guide is the fact that there are already many applicable standards and guides. The relevant information needs to be selected and expressed in a succinct manner, with appropriate pointers to references. The DPDG aims to compile the most applicable parts of earlier guides into a single document outlining the typical development process for Earth Science data products. Standards and best practices formally endorsed by the Earth Science Data and Information System (ESDIS) Standards Office (ESO), outputs from ESDSWGs (e.g., Dataset Interoperability Working Group, and Data Quality Working Group), and recommendations from Distributed Active Archive Centers and data producers are emphasized.

Ramapriyan, Hampapuram↗

NASA NTRS1994

A new systems engineering approach to streamlined science and mission operations for the Far Ultraviolet Spectroscopic Explorer (FUSE)

The Mission Operations and Data Systems Directorate (MO&DSD, Code 500), the Space Sciences Directorate (Code 600), and the Flight Projects Directorate (Code 400) have developed a new approach to combine the science and mission operations for the FUSE mission. FUSE, the last of the Delta-class Explorer missions, will obtain high resolution far ultraviolet spectra (910 - 1220 A) of stellar and extragalactic sources to study the evolution of galaxies and conditions in the early universe. FUSE will be launched in 2000 into a 24-hour highly eccentric orbit. Science operations will be conducted in real time for 16-18 hours per day, in a manner similar to the operations performed today for the International Ultraviolet Explorer. In a radical departure from previous missions, the operations concept combines spacecraft and science operations and data processing functions in a single facility to be housed in the Laboratory for Astronomy and Solar Physics (Code 680). A small missions operations team will provide the spacecraft control, telescope operations and data handling functions in a facility designated as the Science and Mission Operations Center (SMOC). This approach will utilize the Transportable Payload Operations Control Center (TPOCC) architecture for both spacecraft and instrument commanding. Other concepts of integrated operations being developed by the Code 500 Renaissance Project will also be employed for the FUSE SMOC. The primary objective of this approach is to reduce development and mission operations costs. The operations concept, integration of mission and science operations, and extensive use of existing hardware and software tools will decrease both development and operations costs extensively. This paper describes the FUSE operations concept, discusses the systems engineering approach used for its development, and the software, hardware and management tools that will make its implementation feasible.

Butler, Madeline J.↗

NASA NTRSDate not supplied

VEDA: Visualization, Exploration, & Data Analysis

NASA's Visualization, Exploration, and Data Analysis (VEDA) project is an open-source science cyberinfrastructure for data processing, visualization, exploration, and geographic information systems (GIS) capabilities. Developed collaboratively and mostly reusing existing open-source components, VEDA consolidates GIS delivery mechanisms, processing platforms, analysis services, and visualization tools and provides an ecosystem of open tools for addressing Earth science research and application needs through the public-facing VEDA Dashboard. In this presentation, Dr. Freitag will provide an overview of VEDA and how it can potentially serve the AOS community.

Brian Freitag↗

NASA NTRS2002

Early data processing experiences at the AIRS team leader science computing facility

Explore the source record for details and available documents.

AIRS aqua EOS DAAC AMSU HSB↗

NASA NTRS2007

Using Selection Pressure as an Asset to Develop Reusable, Adaptable Software Systems

The Goddard Earth Sciences Data and Information Services Center (GES DISC) at NASA has over the years developed and honed several reusable architectural components for supporting large-scale data centers with a large customer base. These include a processing system (S4PM) and an archive system (S4PA) based upon a workflow engine called the Simple Scalable Script based Science Processor (S4P) and an online data visualization and analysis system (Giovanni). These subsystems are currently reused internally in a variety of combinations to implement customized data management on behalf of instrument science teams and other science investigators. Some of these subsystems (S4P and S4PM) have also been reused by other data centers for operational science processing. Our experience has been that development and utilization of robust interoperable and reusable software systems can actually flourish in environments defined by heterogeneous commodity hardware systems the emphasis on value-added customer service and the continual goal for achieving higher cost efficiencies. The repeated internal reuse that is fostered by such an environment encourages and even forces changes to the software that make it more reusable and adaptable. Allowing and even encouraging such selective pressures to software development has been a key factor In the success of S4P and S4PM which are now available to the open source community under the NASA Open source Agreement

Berrick, Stephen↗

NASA NTRS2026

QS ATMS Calibration Data Book Government Version: Pre-Launch Version

The Advanced Technology Microwave Sounder (ATMS) SN301 is an instrument onboard the QuickSounder Satellite System, QuickSounder mission (renamed QS). Throughout this document the mission may be referred to as QuickSounder or QS. The purpose of this document is to provide information needed to interpret and process QS ATMS science data. The contents of the data book incorporate results from pre-launch subsystem-level and instrument-level testing as well as post-launch testing performed during instrument commissioning. Major milestones include instrument thermal vacuum (TVAC) testing in May 2024 and observatory TVAC testing in 2026. QS launch will be no earlier than Feb. 15, 2027.

Microwave Radiometer↗

NASA NTRS2014

Architectures Toward Reusable Science Data Systems

Data Processing↗

NASA NTRS2000

Preliminary Operational Results of the TDRSS Onboard Navigation System (TONS) for the Terra Mission

The Earth Observing System Terra spacecraft was launched on December 18, 1999, to provide data for the characterization of the terrestrial and oceanic surfaces, clouds, radiation, aerosols, and radiative balance. The Tracking and Data Relay Satellite System (TDRSS) Onboard Navigation System (ONS) (TONS) flying on Terra provides the spacecraft with an operational real-time navigation solution. TONS is a passive system that makes judicious use of Terra's communication and computer subsystems. An objective of the ONS developed by NASA's Goddard Space Flight Center (GSFC) Guidance, Navigation and Control Center is to provide autonomous navigation with minimal power, weight, and volume impact on the user spacecraft. TONS relies on extracting tracking measurements onboard from a TDRSS forward-link communication signal and processing these measurements in an onboard extended Kalman filter to estimate Terra's current state. Terra is the first NASA low Earth orbiting mission to fly autonomous navigation which produces accurate results. The science orbital accuracy requirements for Terra are 150 meters (m) (3sigma) per axis with a goal of 5m (1 sigma) RSS which TONS is expected to meet. The TONS solutions are telemetered in real-time to the mission scientists along with their science data for immediate processing. Once set in the operational mode, TONS eliminates the need for ground orbit determination and allows for a smooth flow from the spacecraft telemetry to planning products for the mission team. This paper will present the preliminary results of the operational TONS solution available from Terra.

Gramling, Cheryl↗

NASA NTRS2018

Analysis and Review of NASA Earth Science Metadata: How Automation Plays a Role

The Analysis and Review of the Common Metadata Repository (CMR ARC) Team reviews all EOSDIS metadata. The team’s objective is to achieve consistency, correctness, and completeness for all metadata records in the CMR, as well as improve the discoverability of NASA's Earth Science data within the CMR framework. This work is currently being completed at Marshall Space Flight Center. CMR makes a single discovery point possible for NASA's Earth Science data users. The CMR team, in collaboration with three other core metadata teams, contributes to the stewardship of NASA's Earth Science data through a process of continual curation and the ongoing development of the Unified Metadata Model (UMM). A key tool now used in the curation process, referred to as the NASA CMR Dashboard, is an online curation dashboard developed in collaboration with software development company, Element 84. This tool facilitates the review of Earth Science metadata records and subsequent stakeholder collaboration on the resolution of identified issues. A key capability of the new tool is a suite of automated compliance checks written in Python 3.6 that verify the integrity of various metadata elements across multiple standards.

Staton, Patrick↗

NASA NTRS2010

Kepler Science Operations Center Pipeline Framework

The Kepler mission is designed to continuously monitor up to 170,000 stars at a 30 minute cadence for 3.5 years searching for Earth-size planets. The data are processed at the Science Operations Center (SOC) at NASA Ames Research Center. Because of the large volume of data and the memory and CPU-intensive nature of the analysis, significant computing hardware is required. We have developed generic pipeline framework software that is used to distribute and synchronize the processing across a cluster of CPUs and to manage the resulting products. The framework is written in Java and is therefore platform-independent, and scales from a single, standalone workstation (for development and research on small data sets) to a full cluster of homogeneous or heterogeneous hardware with minimal configuration changes. A plug-in architecture provides customized control of the unit of work without the need to modify the framework itself. Distributed transaction services provide for atomic storage of pipeline products for a unit of work across a relational database and the custom Kepler DB. Generic parameter management and data accountability services are provided to record the parameter values, software versions, and other meta-data used for each pipeline execution. A graphical console allows for the configuration, execution, and monitoring of pipelines. An alert and metrics subsystem is used to monitor the health and performance of the pipeline. The framework was developed for the Kepler project based on Kepler requirements, but the framework itself is generic and could be used for a variety of applications where these features are needed.

Klaus, Todd C.↗

NASA NTRS1976

Caltech/JPL Conference on Image Processing Technology, Data Sources and Software for Commercial and Scientific Applications

Recent advances in image processing and new applications are presented to the user community to stimulate the development and transfer of this technology to industrial and commercial applications. The Proceedings contains 37 papers and abstracts, including many illustrations (some in color) and provides a single reference source for the user community regarding the ordering and obtaining of NASA-developed image-processing software and science data.

Redmann, G. H.↗

NASA NTRS2019

Bringing Analysis Closer to Data: Developing a Visualization Tool for L2 Earth Science Satellite Data

Earth Science satellite missions provide a unique opportunity for scientists to visualize complex and multifaceted observations projected geospatially across maps of the Earth. While visualization tools can help scientists comprehend, analyze, and share data, visualizing Level-2 Earth Sciences data poses its own specific set of challenges. Since the geospatial information in Level-2 data files is stored as independent variables, the plotting process involves matching dimensional information from latitude and longitude with a desired variable. Variables are stored in different ways across various Earth Science data file formats, which complicates the process of extracting data and plotting variables from a given file without requiring extensive user input and prerequisite familiarity with the file type variable structure. In coordination with NASA’s Goddard Earth Sciences Data Information Services Center (GES DISC), the team developed a Level-2 Earth Science data visualization tool that aims to address some of the complexities associated with plotting Level-2 data. This tool offers command-line and user interface support for file and variable selection to accommodate varying use cases and degrees of user familiarity with the structure of a given file. The visualization tool is written in Python 3 and utilizes a modular approach to facilitate continued expansion and reuse. In addressing some common complications involved in plotting Level-2 Earth Sciences data, the tool aims to help to link the process of analysis more directly with data acquisition and visualization, bringing analysis closer to data across levels of processing.

Li, Angela W.↗

NASA NTRS2012

The Kepler End-to-End Data Pipeline: From Photons to Far Away Worlds

The Kepler mission is described in overview and the Kepler technique for discovering exoplanets is discussed. The design and implementation of the Kepler spacecraft, tracing the data path from photons entering the telescope aperture through raw observation data transmitted to the ground operations team is described. The technical challenges of operating a large aperture photometer with an unprecedented 95 million pixel detector are addressed as well as the onboard technique for processing and reducing the large volume of data produced by the Kepler photometer. The technique and challenge of day-to-day mission operations that result in a very high percentage of time on target is discussed. This includes the day to day process for monitoring and managing the health of the spacecraft, the annual process for maintaining sun on the solar arrays while still keeping the telescope pointed at the fixed science target, the process for safely but rapidly returning to science operations after a spacecraft initiated safing event and the long term anomaly resolution process.The ground data processing pipeline, from the point that science data is received on the ground to the presentation of preliminary planetary candidates and supporting data to the science team for further evaluation is discussed. Ground management, control, exchange and storage of Kepler's large and growing data set is discussed as well as the process and techniques for removing noise sources and applying calibrations to intermediate data products.

data archiving↗

NASA NTRS2014

Prototype and Metrics for Data Processing Chain Components of IPM

This presentation lays out the evolution of the Intelligent Payload Module (IPM) vision given that the HyspIRI mission has been delayed. It shows that there has been a focus on airborne vehcile and unmanned aerial systems to further develop the IPM functionality. This of course does not preclude use of the IPM for space missions but provides alternate paths to continue the concept of improved onboard processing for low latency users of science data products.

Onboard Processing↗

DOE OSTI2026

The LCLStream Ecosystem for Multi-Institutional Dataset Exploration

We describe a new end-to-end experimental data streaming framework designed from the ground up to support new types of applications – AI training, extremely high-rate X-ray time-of-flight analysis, crystal structure determination with distributed processing, and custom data science applications and visualizers yet to be created. Throughout, we use design choices merging cloud microservices with traditional HPC batch execution models for security and flexibility. This project makes a unique contribution to the DOE Integrated Research Infrastructure (IRI) landscape. By creating a flexible, API-driven data request service, we address a significant need for high-speed data streaming sources for the X-ray science data analysis community. With the combination of data request API, mutual authentication web security framework, job queue system, high-rate data buffer, and complementary nature to facility infrastructure, the LCLStreamer framework has prototyped and implemented several new paradigms critical for future generation experiments.

Rogers, David [ORNL] (ORCID:0000000251871768)↗