Engineering PapersSearch

SEARCH · Engineering Papers

Results for “ESDIS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

GES DISC Data Recipes in Jupyter Notebooks

The Earth Science Data and Information System (ESDIS) Project manages twelve Distributed Active Archive Centers (DAACs) which are geographically dispersed across the United States. The DAACs are responsible for ingesting, processing, archiving, and distributing Earth science data produced from various sources (satellites, aircraft, field measurements, etc.). In response to projections of an exponential increase in data production, there has been a recent effort to prototype various DAAC activities in the cloud computing environment. This, in turn, led to the creation of an initiative, called the Cloud Analysis Toolkit to Enable Earth Science (CATEES), to develop a Python software package in order to transition Earth science data processing to the cloud. This project, in particular, supports CATEES and has two primary goals. One, to transition data recipes created by the Goddard Earth Science Data and Information Service Center (GES DISC) into an interactive and educational environment using JupyterNotebooks. Two, to acclimate Earth scientists to cloud computing. To accomplish these goals, we create JupyterNotebooks to compartmentalize the different steps of data analysis and help users obtain and parse data from the command line. We also develop a Docker container, comprised of Jupyter Notebooks, Python dependencies, and command line tools, and configure it into an easy-to-deploy package. The end result is an end-to-end product that simulates the use case of end users working in the cloud computing environment.

discoverability

New Metadata Capabilities within NASA's Common Metadata Repository (CMR)

This talk will convey the new capabilities and features of the UMM-Variables and the UMM-Services metadata model within the Common Metadata Repository (CMR) and how community engagement through the ESDIS Standards Office (ESO) review shaped the new versions of the models. The Unified Metadata Model (UMM) provides a common metadata model to unify legacy systems (i.e. GCMD (Global Change Master Directory), ECHO (Earth Observing System (EOS) Clearinghouse)) with new systems (i.e. CMR). The rationale and migration process of the Service Entry Resource Format (SERF) to the UMM-S will also be conveyed. The talk will conclude with discussing issues and lessons learned from the review process and how future reviews will be conducted to ensure a more targeted, meaningful review with a faster turn-around time for triage and implementation.

Earth Observing System

Generalizing a Data Analysis Pipeline in the Cloud to Handle Diverse Use Cases in NASA's EOSDIS

NASA's Earth Observing System Data and Information System (EOSDIS) is tasked with archiving and distributing Earth Observation data across a range of disciplines, including atmospheric science, oceanography, land processes, natural hazards, solar radiance and even socioeconomic aspects relating to the environment. Driven by rapidly rising data volumes, EOSDIS is migrating to a cloud computing based archive over the next few years. Although this simplifies data management somewhat, the main aim is to provide the data in an environment where end users can bring their analysis to the data rather than attempting to download and manage ever-increasing volumes. To that end, a cloud-based analysis platform is being constructed to enable data transformations, analyses and visualization without egressing the data from the cloud. In this endeavor, we expect a wide variety of users, algorithms and use cases. Consequently, the architecture of this cloud analytics platform is expressly designed to be based on open services, thus fostering an ecosystem that enables the efficient combination of common components with data-specific or analysis-specific components. Reviewed and approved by Andrew Mitchell, ESDIS project manager.

Cloud computing

Global Change Master Directory (GCMD) Keyword Management Process and Lifecycle

The Global Change Master Directory (GCMD) keywords are a hierarchical set of controlled vocabulary covering the Earth science disciplines that have been evolving for over 25 years. The process for how these keywords have been curated and reviewed has also evolved. This presentation will convey the process for reviewing and approving the GCMD keywords, including fast track and yearly reviews through the ESDIS Standards Office, and how Earth science users can influence keyword additions and modifications. The presentation will also highlight how keywords facilitate the discovery of EOSDIS data and services and how other organizations are using the GCMD keywords.

EOSDIS

HDF5 Roadmap 2019-2020

In this talk we will give an overview of the new features of the upcoming HDF5 release 1.12.0, and outline the HDF5 roadmap for the next year. We will demonstrate new open source file drivers to access HDF5 files via Amazon Simple Storage Service (Amazon S3) and on Hadoop Distributed File system (HDFS). We will use this presentation to get feedback on the HDF5 roadmap from the ESDIS users and application developers.

Cloud

NASA Earthdata Cloud

ESDIS is representing NASA at the Nov 21, 2019 NOAA NESDIS Cloud Summit. Briefing material is to summarize the NASA Enterprise Managed Cloud Computing Program and the NASA Enterprise covering high level services, capabilities, architecture, and lessons learned.

McInerney, Mark

Standards and Best Practices - Two NASA Examples

Formal international standards as well as promotion of community or recommended practices have their place in ensuring "FAIRness" of data. Data management in NASA's Earth Observation System Data and Information System (EOSDIS) has benefited from both of these avenues to a significant extent. The purpose of this paper is to present one example of each of these, which promote (re)usability. The first is an ISO standard for specifying preservation content from Earth observation missions. The work on this started in 2011, informally within the Earth Science Information Partners (ESIP) in the US, while the European Space Agency (ESA) was leading an effort on Long-Term Data Preservation (LTDP). Resulting from the ESIP discussions was NASA's Preservation Content Specification, which was applied in 2012 as a requirement for NASA's new missions. ESA's Preserved Data Set Content (PDSC) document was codified into a document adopted by the Committee on Earth Observation Satellites (CEOS). It was recognized that it would be useful to combine PCS and PDSC into an ISO standard to ensure consistency in data preservation on a broader international scale. This standard, numbered ISO 19165-2 has been under development since mid-2017. The second is an example of developing recommendations for "best practices" within more limited (still fairly broad) communities. A Data Product Developers' Guide (DPDG) is currently being developed by one of NASA's Earth Science Data System Working Groups (ESDSWGs). It is for use by developers of products to be derived from Earth observation data to improve product (re)usability. One of the challenges in developing the guide is the fact that there are already many applicable standards and guides. The relevant information needs to be selected and expressed in a succinct manner, with appropriate pointers to references. The DPDG aims to compile the most applicable parts of earlier guides into a single document outlining the typical development process for Earth Science data products. Standards and best practices formally endorsed by the Earth Science Data and Information System (ESDIS) Standards Office (ESO), outputs from ESDSWGs (e.g., Dataset Interoperability Working Group, and Data Quality Working Group), and recommendations from Distributed Active Archive Centers and data producers are emphasized.

Ramapriyan, Hampapuram

TPSAS-NF1676L-32053-DND

The Atmospheric Science Data Center (ASDC) offers Earth Science data sets created from satellite measurements, modeling, and field experiments enabling scientists, educators, and the public to study earth and its atmosphere. NASA ESDIS is working towards making Earth Science data and tools cloud-ready. In an effort to align itself with these efforts, the ASDC is transitioning its existing web based services, tools and applications to RESTful APIs, containerized as microservices to leverage scalability and efficiency of an on-premise cloud environment.

Makhan L Virdi

A User-Focused Renovation of CERES Metadata

Production software and public data products for Clouds and the Earth’s Radiant Energy System (CERES) continue to evolve as the project extends its climate data record. The data management team for CERES is currently undertaking major renovations of both code and data products, the latter of which is, of course, in service of improving user experience. A major mode of CERES’ data product improvement is in renovating products’ metadata. Metadata standards have evolved since CERES began producing its data products in 2000. In its twentieth year, CERES essentially asked the question: how would the project design its data products if it could start all over again? With forthcoming editions, this rebirth will be realized. CERES has redesigned its metadata standards to best position itself for data discoverability. The project has used the latest standards being developed in NASA’s Earth Science Data and Information Systems (ESDIS) Project’s Unified Metadata Model (UMM) documentation; collaborated with the Atmospheric Science Data Center (ASDC) to ensure compliance with Common Metadata Repository compatibility, and continued compliance with Climate and Forecast (CF) Conventions. In doing so, the team created its own, internal document for proper metadata creation and metadata verification software that is deployed prior to all code deliveries. This presentation will discuss this redesign process, as well as needs met and those that are still outstanding in the search for an improved user experience with CERES data products.

Kathleen Dejwakh

TRUST, Trustworthiness and EOSDIS

In recent years there has been considerable attention by the international scientific research and applications community to ensure high quality of data and information management. The terms FAIR (Findable, Accessible, Interoperable, Reusable) data, TRUST (Transparency, Responsibility, User Community, Sustainability, and Technology) principles, and CARE (Collective Benefit, Authority to Control, Responsibility, and Ethics) principles have come into vogue during the last decade. NASA has been managing data and information for over 60 years. NASA’s Earth Observing System Data and Information System (EOSDIS) has been in operation for over 25 years, managing most of NASA’s Earth science data. Trustworthiness is a goal that NASA has always strived to achieve or exceed, because it: enables the success of any NASA science mission; inspires general science research and applications; justifies the cost of operations; contributes to the value of NASA’s Open Data Policy; and influences the long term, historical view for the data collection. Given the recent growth of interest in TRUST principles, it is useful to assess and show how NASA’s attention to trustworthiness maps into those principles. This presentation addresses shows how the various steps that have been taken by the Earth Science Data and Information System (ESDIS) Project in the implementation and evolution of EOSDIS map into the TRUST principles.

Remote Sensing

EED-2 Contract Final Report

This EED-2 contract provides for development and sustaining engineering of software and hardware systems that provide science data management for the ESDIS Project. A major activity under this contract will be for evolution and development engineering of the EOSDIS Core System (ECS), Earthdata platform, Common Metadata Repository (CMR), NASA-Compliant General Application Platform (NGAP) cloud environment, and other EOSDIS elements that provide the common capabilities and infrastructure of EOSDIS.

EOSDIS

Earth Observation Data Provenance for Future Climate Research - Requirements and Challenges

Observations and measurements of the Earth’s environment have been collected from space since the 1960’s. Flight Projects, airborne and field campaigns have developed and operated multi-year missions with global observing instruments, and Principle Investigator Science Teams have developed algorithms, calibrated, and derived a wide variety of Earth system environmental parameters. These data are archived and distributed for research purposes by NASA’s ESDIS Project and Distributed Active Archive Centers (DAACs). They are expected to be an important basis for Earth science and climate change research extending well beyond their observation times, the Principle Investigators/Science Teams research projects and life of the Flight Projects.

John Moses

FAIR-ness Assessment of NASA’s Earth Observation System Data and Information System (EOSDIS)

This presentation addresses the challenge of evaluating a multi-disciplinary institutional network of data repositories in operation since 1994 against the relatively recent criteria that constitute FAIR (Findable, Accessible, Interoperable, Reusable) data. NASA’s Earth Observation System Data and Information System (EOSDIS), with its 12 discipline-based Distributed Active Archive Centers (DAACs), preceded the definition and popularization of FAIR by over two decades. An assessment is very useful to describe how well the FAIR principles are met and to identify any improvements needed. In 2020, A “self-assessment” of EOSDIS and DAACs was performed by the ESDIS Project staff and the DAACs from the points of view of human actionability and machine actionability. More recently, a draft of a Science Mission Directorate (SMP) Program Directive (SPD-41a) has been released by NASA Headquarters for comment, where it is recommended that all SMD-funded data should follow the FAIR principles. This presentation is timely to initiate community discussion within the Information Quality Cluster (IQC) of the Earth Science Information Partners (ESIP) and help strategize and develop implementation guidelines for EOSDIS and DAACs to conform to FAIR principles.

Remote sensing

What is LAADS DAAC? How to Download Data from NASA's Source for Cloud and Aerosol Data

The Level-1 and Atmosphere Archive Distribution System (LAADS) Distributed Active Archive Center (DAAC) primarily archives and distributes data on clouds and aerosols in Earth’s atmosphere in addition to providing Level 0 and 1 Moderate Imaging Radiometer Suite (MODIS), Visible Infrared Imaging Radiometer Suite (VIIRS) and airborne data as well as data from the European Space Agency’s Sea and Land Surface Temperature Radiometer (SLSTR) and the Ocean and Land Colour Instrument (OLCI). Secondarily, LAADS DAAC serves as a backup source for MODIS and VIIRS land products. As one of twelve DAACS supported by NASA’s Earth Science Data and Information System (ESDIS), LAADS DAAC provides a gateway to data through multiple tools. The newest, View Data, allows users to preview data sets before downloading or applying any post-processing transformations. Learn about how to access LAADS DAAC data, get up-to-date information about LAADS DAAC's migration to hosting data in the cloud, and discover how to use the View Data tool.

Tassia Owen

LAADS DAAC Migrates to the Cloud: Lessons Learned from Communicating About Earth Science Data on the Cloud

The Level-1 and Atmosphere Archive Distribution System (LAADS) Distributed Active Archive Center (DAAC) is migrating data to the cloud. As one of twelve DAACS supported by NASA’s Earth Science Data and Information System (ESDIS), LAADS is using moving away from on premise data storage facilities to migrating to Amazon Web Services, where the massive archive of data from the Moderate Imaging Spectroradiometer (MODIS) and the Visible Infrared Imaging Radiometer Suite (VIIRS) will be available for download and post-processing transformations online. The migration is happening in three phases and LAADS is concluding its beta testing period. This poster shows the lessons learned from communicating with a select group of users about how to effectively educate data users on using data in the cloud.

Tassia Owen

Open Science for Climate at NASA

The US National Aeronautics and Space Administration (NASA) currently provides more than 75 Petabytes of open access data through NASA’s Earth Science Data and Information System (ESDIS). Many of these datasets are commonly used in WCRP research, including the Global Precipitation Measurement (GPM) mission dataset, the Soil Moisture Active Passive (SMAP) mission dataset, the IceBridge dataset, and the Atmospheric Infrared Sounder (AIRS) dataset. These datasets are essential tools for understanding patterns in precipitation, soil moisture, ice mass changes, atmospheric circulation, and their impacts on climate. NASA is also a leader in the White House OSTP’s “Federal Year of Open Science” in 2023, promoting open access and open source solutions.

Climate Data

Assessing the Needs of NASA's Near Real-Time Earth Observation Products

"The 2017-2027 Decadal Survey for Earth Science and Applications from Space stated that NASA's Earth Science with planned implementation of applications provides sustained earth observations for societal benefits [1]. The Decadal Survey indicated that data latency is invaluable for time-sensitive applications including disaster risk reduction, wildland fire carbon emissions quantification, real-time measurements of the state of the hydrologic systems and many more. Data latency refers to the time between earth observation and data products available to users. During the past 13 years, NASA's Land, Atmosphere Near Real-Time Capability for Earth Observing Systems (LANCE) continues to provide free access to earth observation products that are made available much quicker than routine processing allows. The latency of most LANCE data products is Near Real-time (NRT) which is defined as less than three hours from satellite observations [2]. LANCE is managed by the Earth Science Data and Information System (ESDIS) Project at NASA Goddard Space Flight Center [3], and a User Working Group (UWG) is responsible for providing guidance to LANCE. LANCE data are used by direct users and brokers who add value to the data [4]. NASA Earth Applied Sciences Program (ASP) is one of the primary users of LANCE, which collaborates with partner organizations and provides support to scientists to solve problems in applications of earth observations. ASP promotes the use of LANCE NRT data products to demonstrate applications in decision making, facilitates end-user feedback to the science team to improve data products, and provides information on future demands for research. LANCE supports applications that need a rapid response including detecting wildland fires and volcanic eruptions, tracking smoke, ash and dust plumes, monitoring air quality and tracking extreme weather events such as hurricanes, landslides, and floods. To gather feedback regarding the availability, accessibility and actionability of NASA's NRT data products for societal benefit, three surveys and a few discussions with experts involved in the topic within ASP were conducted from the perspective of users. Feedback has been collected from users who are interested in using low latency NASA data within application communities of agriculture, disasters, water resources, health and air quality, ecological conservation, wildland fires and capacity building. Analysis-ready NRT data products in a variety of formats have been mentioned many times in the collected feedback, especially for applied users with little to no experience using research-grade earth observation products. Users prefer to have products that can be easily integrated into their existing workflows and take their analysis to the data. HDF5 is a commonly used data format for research, but typically requires some conversion to a more friendly format for applications and regular use in decision-making. Users prefer the GeoTIFF data format that can be directly ingested into a GIS mapping software and platform for data analysis and visualization. For example, LANCE’s fire, flood, SO2 and Black Marble Nighttime Blue/Yellow Composite data products have been integrated into NASA Disasters Mapping Portal, which is an GIS-based open data portal, for users in the disaster management community. There are 291 LANCE NRT layers available through GIBS and Worldview, where users can download a snapshot in GeoTIFF format. Operational users expect data to be processed as close to the user as possible. The collected feedback indicates that LANCE fire products within 3 hours latency would meet the needs of the wildland fire community. The ideal latency for volcanic application is 10-15 minutes. Users in Volcanic Ash Advisory Centers (VAAC) reported that the first forecast volcanic product should be issued within 75 minutes from the volcano eruption [5]. Overall, for disaster applications, data latency within 3 hours is useful while latency greater than 12 hours is not timely enough for operational use. Capacity building and training are critical for users to be able to access, interpret and use data products and tools for their decision making, especially for applied users with limited experience using earth observation products. LANCE data products have been used in a number of capacity building projects domestically and internationally [6]. As LANCE continues to bring new products into the system, users request training to utilize LANCE new and upcoming data products and capabilities in their applications. Due to the limitation of bandwidth and downstream flow paths, users in some developing countries need tools to select and download data for a specific area of interest instead of bulk downloads. The collected feedback also shows the lack of available SAR satellite low latency data products. The advantages of SAR to monitor conditions and changes on the ground through darkness, clouds, volcanic ash, and other atmospheric conditions, are appealing to low latency users. For example, terabytes of low latency but cloudy optical images are not helpful in rapidly identifying the extent of flood or fire impacts. LANCE could be complemented with low latency measurements via the upcoming NASA-ISRO Synthetic Aperture Radar (NISAR) mission [7]. Requests for higher spatial resolution products are expressed. A user from the wildland fire management community reported that products with 30-m spatial resolution could be used to detect small fires. The 30-m Landsat OLI fire data is now part of NASA’s Fire Information for Resource Management System (FIRMS) US/Canada [8]. Within the open and free NASA resources, LANCE disseminates NRT data products in a manner that allows them to be accessible and understandable to both scientific and applied users. In many application areas, latency plays an important or even decisive role where low latency earth observations help people to observe areas of interest, detect and track changes in the environment and make timely decisions. NASA’s Earth Applied Sciences Program promotes the use of LANCE NRT products and builds a bridge between application users and research teams. The collected feedback indicates data latency within 3 hours is useful for most of the applications, and shows the needs of user-friendly, analysis-ready products, and requests training on LANCE’s new and upcoming data products. User feedback has been provided to LANCE UWG for guidance and recommendations, and for translating findings into something actionable.

Tian Yao

NASA’s Data Preservation Strategies for Ensuring Ongoing Use and Access: Leveraging the Capabilities at the NASA DAACs

As NASA prepares for the follow on to EOS through the deployment of many new space and airborne-based assets envisioned under the ESO initiative, the need for ensuring user access to the very large and diverse archive of past missions remains a critical endeavor for current and future research. The ESDIS team has been working for many years on developing strategies and implementations that will make the linkage between the data held in our archives and the emerging missions possible. The focus on how best to preserve these past data has proven to be a collaborative effort, and one that may well benefit from the adoption of innovative technical tools that enable organizational efficiencies. This poster reveals both aspects of NASA’s EOS data repository strategy by discussing the collaborative community efforts of development of ISO 19165-2 – an international standard supported by a team from many countries around the world, the use by EOSDIS DAACs of a Preservation Content Specification, and ongoing work with mission teams to ensure mission data and all associated information are preserved and available for current and future use.

Francis Lindsay