Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Amazon Web Services”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

GC31G-1182: Opennex, a Private-Public Partnership in Support of the National Climate Assessment

The NASA Earth Exchange (NEX) is a collaborative computing platform that has been developed with the objective of bringing scientists together with the software tools, massive global datasets, and supercomputing resources necessary to accelerate research in Earth systems science and global change. NEX is funded as an enabling tool for sustaining the national climate assessment. Over the past five years, researchers have used the NEX platform and produced a number of data sets highly relevant to the National Climate Assessment. These include high-resolution climate projections using different downscaling techniques and trends in historical climate from satellite data. To enable a broader community in exploiting the above datasets, the NEX team partnered with public cloud providers to create the OpenNEX platform. OpenNEX provides ready access to NEX data holdings on a number of public cloud platforms along with pertinent analysis tools and workflows in the form of Machine Images and Docker Containers, lectures and tutorials by experts. We will showcase some of the applications of OpenNEX data and tools by the community on Amazon Web Services, Google Cloud and the NEX Sandbox.

datasets↗

Geonex: A NASA-NOAA Collaboration for Producing Land Surface Products from Geostationary Sensors Using Cloud Computing

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement.Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GEONEX pipeline including choice CPUs, storage media, and automation. We estimate the cost of deploying GEONEX to be $400 - 750 a month for processing data (every 30 minutes) and producing products over the conterminous US. For products such as Fire, latency can be as little as 10 minutes from the time of data acquisition.

Geostationary↗

GEONEX: Webpage to Display NASA-NOAA Collaboration of Producing Land Surface Products from Geostationary Sensors

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement.In order to better introduce the GEONEX products to the science community, we set up a simple webpage (www.geonex.org) to describe the background and the motivation of the project, the algorithms used in deriving the products, and user manuals to the data files. We will also update the status of the data processing, in particular the near-real-time products, on the website and provide links (in text or json files) to the latest datasets.

geostationary↗

Geonex: Land Surface Monitoring from a New Generation of Geostationary Sensors

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement. Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GEONEX pipeline including choice CPUs, storage media, and automation. By making the GEONEX pipeline available on the cloud, we hope to engage a broad community of Earth scientists from around the world in utilizing this new source of data for Earth monitoring.

Nemani, Ramakrishna R.↗

Earth Observations from Geostationary Satellites

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GeoNEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GeoNEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement. Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GeoNEX pipeline including choice CPUs, storage media, and automation. By making the GEONEX pipeline available on the cloud, we hope to engage a broad community of Earth scientists from around the world in utilizing this new source of data for Earth monitoring.

Earth↗

GeoNEX: Land Monitoring from a New Generation of Geostationary Sensors

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement. Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GEONEX pipeline including choice CPUs, storage media, and automation. By making the GEONEX pipeline available on the cloud, we hope to engage a broad community of Earth scientists from around the world in utilizing this new source of data for Earth monitoring.

GeoNEX↗

Forming Aggregations using Virtual Sharding: Lessons Learned from Simple Scalable Storage (S3)

Data aggregation is the ability to combine separate datasets to form a single new logical dataset provides users with a powerful abstraction. The advantage of an aggregate dataset is that the users are freed from having to understand, and incorporate into their workflow, knowledge about the (ad hoc) organization of the constituent datasets. However, aggregating large numbers of files can be computationally complex with data server systems performing many repetitive operations. As part of the authors work on subsetting data stored on Amazon Web Service (AWS) Simple Storage Service (S3), we developed technology to read portions of otherwise monolithic data files. This enables the formation of virtual shards for user in subsetting data stored in HDF5 (hierarchical data format, version 5) files. This same tool can be used to form aggregations that combine data stored in many HDF5 files when those files are stored on S3. The nature of the virtual sharding and the algorithm that exploits it for subsetting is such that it can also be used for aggregation with the need for many of the repetitive operations required by the per file aggregation techniques. We will present timing information that demonstrates the flexibility of this approach. However, the lessons learned is that while this is a useful result in and of itself, these very same techniques can be applied in other contexts where data are stored in services and on media other than S3. For example, this same technique can be applied to data stored on spinning disk. Pushing the envelope for S3 forced a reexamination of our data access techniques which lead to unexpected positive benefits.

Gallagher, James↗

Jumping the Queue: From NASA to the Commercial Cloud

NASA's High-End Computing Capability (HECC) Project has made it possible for its users to run on commercial cloud resources in a seamless way. In the first of three phases, we implemented a pilot project for a few users, enabling them to “jump the queue” and burst jobs from the HECC environment to Amazon Web Services (AWS). By using GPU-accelerated nodes at AWS, the users were able to make significant advances in their research. The second phase of the project made AWS access available to all HECC users and added accounting to make users responsible for cloud charges. We are also enabling export-controlled work through the use of AWS GovCloud. In the third phase, we will add web-based mechanisms to permit non-HECC users to access cloud resources for their HPC projects.

Hood, Robert↗

Results from the ASTERIA CubeSat Extended Mission Experiments

Over the past two years, JPL has used the ASTERIA (Arcsecond Space Telescope Enabling Research In Astrophysics) CubeSat as an in-flight test platform during extended missions. ASTERIA successfully completed its prime mission in early 2018, and continued to operate in low Earth orbit (LEO) for an additional twenty months. This paper describes demonstrations that were performed on the spacecraft and on the ground-based testbed during the extended mission. These demonstrations fall into three categories: Autonomy technology maturation, hardware characterization, and science discovery. Autonomy technology maturation supported three development efforts. The first shifted the spacecraft commanding paradigm from time-based sequences to Task Networks (tasknets), which allow simpler commanding and more robust onboard execution. The second demonstrated onboard orbit determination in Low Earth Orbit (LEO) without GPS. This activity used a fully-independent means of spacecraft orbit determination for Earth orbiters using only passive imaging. The third technology provided in situ hardware health state estimation using a model-based reasoning technique. These three technologies were demonstrated either in flight or on the testbed individually, and then were combined to demonstrate the capability to perform autonomous navigation on board without ground intervention, even in the presence of anomalies. Hardware characterization involved both onboard and ground-based activities. On board, nonstandard attitude control modes were commanded to characterize the spacecraft pointing jitter as a function of target brightness, reaction wheel speed, controller gain, and the number of guide stars. The results provide insights into the contribution of jitter to the ASTERIA photometry and inform the feasibility of future astrophysics small satellite missions for which jitter control is an enabling technology. On the ground, the ASTERIA Operations Team coordinated with Amazon Web Services (AWS) to configure their new ground stations to communicate with ASTERIA to prove out their viability. ASTERIA used AWS ground stations for nominal operations for the last four months of the mission. Finally, ASTERIA continued to perform exoplanet science as the spacecraft was well-suited to execute long-term monitoring of stars such as alpha Centauri to search for small transiting planets. The science team also imaged a number of interesting objects including a comet, an asteroid, cities at night, and the moon, and coordinated with other projects on Targets of Opportunity for follow-up confirmations and co-observations. Throughout the prime and the extended missions, the ASTERIA spacecraft proved to be a mighty platform that “will go into history as an innovative milestone.”[1 - Zurbuchen]

Doran, Patrick↗

Characterizing Wildfires in Western US.: A Cloud-based Case Study for Interdisciplinary Research using NASA Resources

This presentation will demonstrate a case study of interdisciplinary research done in the Amazon Web Services (AWS) cloud platform, in addition to in the local machine. We conduct data analysis next to data by leveraging various cloud-based data in NASA Earthdata Cloud, which are distributed by different missions/NASA Distributed Active Archive Centers (DAACs), and cloud computing resources at NASA. For instance, we directly access multiple datasets stored in the AWS Simple Storage Service (S3) buckets using a Python Jupyter notebook through a JupyterHub interface hosted in AWS (without having to download data), and conduct data analysis next to data in the cloud. We will also show how to share the research results following Open Source policy. This case study characterizes the change in wildfire events in the western United States during the past 20 years. In particular, we focus on the wildfires in California in 2021, one of the most severe wildfire years occurring in the most recent 20 years in California. We will analyze the possible causes of wildfires, such as drought conditions and climate variability, and examine the impacts of wildfires on air quality and atmospheric composition, and on land cover. We will examine the data distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), including aerosols and meteorological data from the NASA Modern-Era Retrospective analysis for Research and Applications version 2 (MERRA-2), precipitation from the Global Precipitation Measurement (GPM) and Global Precipitation Climate Project (GPCP), and aerosol index from Ozone Monitoring Instrument (OMI). We also utilize the data distributed by the Physical Oceanography (PO) DAAC, such as Sea Surface Temperature (SST) data from the Group for High Resolution Sea Surface Temperature (GHRSST), and the data distributed by Land Processes (LP) DAAC, such as Normalized Difference Vegetation Index (NDVI).

Xiaohua Pan↗

LAADS DAAC Migrates to the Cloud: Lessons Learned from Communicating About Earth Science Data on the Cloud

The Level-1 and Atmosphere Archive Distribution System (LAADS) Distributed Active Archive Center (DAAC) is migrating data to the cloud. As one of twelve DAACS supported by NASA’s Earth Science Data and Information System (ESDIS), LAADS is using moving away from on premise data storage facilities to migrating to Amazon Web Services, where the massive archive of data from the Moderate Imaging Spectroradiometer (MODIS) and the Visible Infrared Imaging Radiometer Suite (VIIRS) will be available for download and post-processing transformations online. The migration is happening in three phases and LAADS is concluding its beta testing period. This poster shows the lessons learned from communicating with a select group of users about how to effectively educate data users on using data in the cloud.

Tassia Owen↗

NASA GEOS Composition Forecast System, GEOS-CF

NASA's Global Modeling and Assimilation Office (GMAO) produces high-resolution analysis and forecasts for weather, aerosols, and air quality. Since 2019, the NASA Global Earth Observing System (GEOS) model provides global near-real-time historical estimates and daily 5-day forecasts of atmospheric composition to the public at unprecedented horizontal resolution of 0.25 degrees (~25 km) from the surface up to the lower mesosphere. This composition forecast system (“GEOS-CF”) combines the operational GEOS weather forecasting model with the state-of-the-science GEOS-Chem chemistry module to deliver detailed analysis of a wide range of air pollutants, including the policy-relevant species such as ozone, carbon monoxide, nitrogen oxides, sulfur dioxide and fine particulate matter (PM2.5). The GEOS-CF is a tool for scientists and the public health community. This presentation will cover 1) an overview of the GEOS-CF modeling framework and data/visualization access, 2) examples of current and future applications to support NASA missions (e.g., a priori for trace gas retrievals by TEMPO, ground-based instrument teams and field campaigns), and 3) research and development activities as the GEOS-CF system continues to evolve to include multi-constituent data assimilation, near-real time emission adjustment estimates, down-scaling methods to urban-scale, and data access on Google Earth Engine, Amazon Web Services, and other platforms to integrate our state-of-the-science air quality information onto platforms used by stakeholders, air quality managers, and the public.

K. Emma Knowland↗

Advancing Open Source Science Initiatives Through Public-Private Partnerships

Collaboration is fundamental to advancing open science within the science community. With the recent developments in technology and research, the establishment of formal partnerships between the private sector and government agencies are needed to bridge the knowledge gaps and expedite the time to actionable science. NASA’s Interagency Implementation and Advanced Concepts Team (IMPACT) seeks to address this challenge by establishing non-reimbursable Space Act Agreements with industry leaders in cloud computing, artificial intelligence (AI) and machine learning. The purpose of these agreements is to advance open source science initiatives in the areas of data discovery, access and use of high value NASA science data sets on the cloud. As well as, jointly work on common research problems to accelerate the development and adoption of new AI technologies. Current success stories include co-locating NASA datasets from multiple science disciplines on one platform using Amazon Web Services Open Data Registry, developing AI Foundation Models for Science with IBM and co-hosting training workshops and tutorials for the science community aimed at providing hands-on experience with using NASA data and AI models on the cloud. In summary, we will present an overview of our partnerships supporting open source science initiatives, describe current activities and lessons learned that may be useful to others considering similar partnerships with the private sector.

Elizabeth Fancher↗

Open Science in Action: The Role of SWxSOC in Expedited Data Release and Cloud-based Data Processing for Heliophysics Missions

NASA's Space Weather Science Operations Center (SWxSOC) is an effort to develop a multi-mission Science Operations Center for the community and specifically Space Weather missions that require data products to be released consistently and quickly. The SWxSOC is committed to Open Science in its approach to software development and data product releases. We are currently supporting the Heliophysics Environmental and Radiation Measurement Experiment Suite (HERMES) that will fly on the Lunar Gateway and the PADRE Small Sat mission. SWxSOC has established an open-source, reusable solution for transitioning data management from on-premises to the cloud, processing data files, and setting up a cloud-based analysis environment to monitor instrument anomalies. Leveraging Amazon Web Services (AWS) and making open-source tools available on GitHub, HERMES exemplifies NASA's commitment to open-source standardization, showcasing the real-world effectiveness of cloud technology in data processing, analysis, and observability. This enhances current operations and ensures faster, standardized deployments for future missions.

hermes↗

Global-to-local air quality forecasts using the NASA GEOS Composition Forecast System

Since 2019, the NASA Global Earth Observing System (GEOS) model has been used to generate global, near-real-time estimates and daily five-day forecasts of atmospheric composition at a horizontal resolution of 0.25 degrees (~25 km) from the surface up to the lower mesosphere. This composition forecast system (“GEOS-CF”) combines the GEOS weather forecasting model with the state-of-the-science GEOS-Chem chemistry module to deliver detailed analysis of a wide range of air pollutants, including the policy-relevant species such as ozone, carbon monoxide, nitrogen oxides, sulfur dioxide and fine particulate matter (PM2.5). Because GEOS-CF includes atmospheric levels up through the stratosphere, this system has been leveraged to support the Tropospheric Emissions: Monitoring of Pollution (TEMPO) satellite mission and provide stratospheric intrusion alerts to ground-based monitoring stations. We will present recent advances to GEOS-CF which target increased computational efficiency and accuracy. These include the incorporation of simplified chemistry mechanisms to accelerate model forecasts, use of model-observation data fusion techniques to provide highly localized forecasts, and assimilation of satellite observations to produce more accurate model analyses. We further discuss our attempts to make these tools publicly available on platforms outside the NASA domain, such as Google Earth Engine and Amazon Web Services with the goal to facilitate the integration of state-of-the-science air quality information onto platforms used by stakeholders, air quality managers, and the public.

Emma Knowland↗

High Performance Access to Archival Data Stored in HDF4 and HDF5 on Cloud Object Stores Without Reformatting the Files

Cloud computing offers numerous advantages for users of extensive Earth science data collections. These benefits encompass direct online access to data files and granules from any location, scalable access supporting parallel computing workflows, and flexible computing tools enabling innovative experimentation with processing techniques. However, older archival file formats designed for distinct computing systems hinder efficient access to decade-long time-series data when compared to data stored in modern cloud-optimized formats like Web Object Stores (WOS), exemplified by Amazon Web Services’ Simple Storage Service (S3). We describe DMR++ (Dataset Metadata Response plus plus), a technology facilitating efficient access to HDF5 (Hierarchical Data Format, version 5) and HDF4 files stored on WOS systems without requiring data reformatting. DMR++ achieves performance comparable to technologies like Zarr while preserving the original file structure, a substantial benefit considering the vast quantity of archival files held by organizations such as NASA. Moreover, DMR++ typically outperforms cloud-optimized versions of HDF5. Essentially an XML (Extensible Markup Language) document usually stored alongside the described data, DMR++ can also be generated on-the-fly but is generally created during data staging to the WOS. Archival files that use HDF4/5 often store large arrays of numerical data. The data in these files is often compressed, typically reducing their size by a factor of four or more. To achieve efficient access to portions of those arrays, they are 'chunked' into smaller sub-arrays, each individually compressed. The chunk size is a compromise, where spinning disks can efficiently access data in smaller chunks while S3 favors larger chunks. A simple optimization of aggregating smaller chunks that are stored adjacently, transferring them in a single access and then individually decompressing them will improve performance. NASA data pose an additional challenge: special Application Programmer Interface (API) libraries are often needed to compute some variables. These libraries are incompatible with WOS environments. Our solution involves storing computed values in the DMR++ document or a companion file, making them accessible like other variables and eliminating the need for specialized APIs. We outline specific optimizations for both satellite grid and swath data stored in HDF4-EOS2 (Earth Observing System).

James Gallagher↗

Open-Source Data Engineering at NASA: CCMC's Approach to Managing Petabyte-Scale Heliophysics Data

The Community Coordinated Modeling Center (CCMC) at NASA Goddard Space Flight Center (GSFC) leads heliophysics research by providing open access to numerous models and their outputs. Our resources are available on-demand and continuously updated with real-time data, covering sun-earth interactions across multiple domains. These domains include coronal, heliosphere, inner and global magnetosphere, ionosphere, thermosphere, and lower atmosphere interactions. Operating in a hybrid environment, CCMC utilizes both self-owned hardware and Amazon Web Services (AWS) cloud infrastructure. Managing petabytes of data across multiple locations necessitates robust data engineering solutions. To address this challenge, CCMC has adopted industry-standard and open-source tools. We use Apache Airflow as our primary data engineering platform, Python for scripting and data processing, and GitLab for version control and CI/CD. Additionally, we employ Kubernetes for containerized services, Grafana and Prometheus for metrics and monitoring, and Terraform and Puppet for reproducible infrastructure as code. This presentation will discuss lessons learned from our data engineering experiences, platforms evaluated but found unsuitable for our scientific data requirements, and specific techniques developed to enhance data transfer speed and reliability. By using these technologies effectively, CCMC continues to advance heliophysics research through efficient data management and open-access modeling.

space weather↗

Analyzing reliability of virtual machine instances with dynamic pricing in the public cloud

This study presents reliability analysis of virtual machine instances in public cloud environments in the face of dynamic pricing. Different from traditional fixed pricing, dynamic pricing allows price to dynamically fluctuate over arbitrary period of time according to external factors such as supply and demand, excess capacity, etc. This pricing option introduces a new type of fault: virtual machine instances may be unexpectedly terminated due to conflicts in the original bid price and the current offered price. This new class of fault under dynamic pricing may be more dominant than traditional faults in cloud computing environments, where resource availability associated with traditional faults is often above 99.9%. To address and understand this new type of fault, we translated two classic reliability metrics, mean time between failures and availability, to the Amazon Web Services spot market using historical price data. We also validated our findings by submitting actual bids in the spot market. We found that overall, our historical analysis and experimental validation lined up well. Based upon these experimental results, we also provided suggestions and techniques to maximize overall reliability of virtual machine instances under dynamic pricing.

Cloud computing↗