Help Us, Zarr. You're Our Only Hope! Migrating Multi-Dimensional Arrays to the Cloud for Data-Heavy Workflows
Explore the source record for details and available documents.
Christine Smit↗
SEARCH · Engineering Papers
Results for “Cloud Migration”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
NASA's Earth Observing Data and Information System - Near-Term Challenges
NASA's Earth Observing System Data and Information System (EOSDIS) has been a central component of the NASA Earth observation program since the 1990's. EOSDIS manages data covering a wide range of Earth science disciplines including cryosphere, land cover change, polar processes, field campaigns, ocean surface, digital elevation, atmosphere dynamics and composition, and inter-disciplinary research, and many others. One of the key components of EOSDIS is a set of twelve discipline-based Distributed Active Archive Centers (DAACs) distributed across the United States. Managed by NASA's Earth Science Data and Information System (ESDIS) Project at Goddard Space Flight Center, these DAACs serve over 3 million users globally. The ESDIS Project provides the infrastructure support for EOSDIS, which includes other components such as the Science Investigator-led Processing systems (SIPS), common metadata and metrics management systems, specialized network systems, standards management, and centralized support for use of commercial cloud capabilities. Given the long-term requirements, and the rapid pace of information technology and changing expectations of the user community, EOSDIS has evolved continually over the past three decades. However, many challenges remain. Challenges addressed in this paper include: growing volume and variety, achieving consistency across a diverse set of data producers, managing information about a large number of datasets, migration to a cloud computing environment, optimizing data discovery and access, incorporating user feedback from a diverse community, keeping metadata updated as data collections grow and age, and ensuring that all the content needed for understanding datasets by future users is identified and preserved.
Remote Sensing↗
Processing NASA Earth Science Data on Nebula Cloud
Three applications were successfully migrated to Nebula, including S4PM, AIRS L1/L2 algorithms, and Giovanni MAPSS. Nebula has some advantages compared with local machines (e.g. performance, cost, scalability, bundling, etc.). Nebula still faces some challenges (e.g. stability, object storage, networking, etc.). Migrating applications to Nebula is feasible but time consuming. Lessons learned from our Nebula experience will benefit future Cloud Computing efforts at GES DISC.
Chen, Aijun↗
Controlling Infrastructure Costs: Right-Sizing the Mission Control Facility
Johnson Space Center's Mission Control Center is a space vehicle, space program agnostic facility. The current operational design is essentially identical to the original facility architecture that was developed and deployed in the mid-90's. In an effort to streamline the support costs of the mission critical facility, the Mission Operations Division (MOD) of Johnson Space Center (JSC) has sponsored an exploratory project to evaluate and inject current state-of-the-practice Information Technology (IT) tools, processes and technology into legacy operations. The general push in the IT industry has been trending towards a data-centric computer infrastructure for the past several years. Organizations facing challenges with facility operations costs are turning to creative solutions combining hardware consolidation, virtualization and remote access to meet and exceed performance, security, and availability requirements. The Operations Technology Facility (OTF) organization at the Johnson Space Center has been chartered to build and evaluate a parallel Mission Control infrastructure, replacing the existing, thick-client distributed computing model and network architecture with a data center model utilizing virtualization to provide the MCC Infrastructure as a Service. The OTF will design a replacement architecture for the Mission Control Facility, leveraging hardware consolidation through the use of blade servers, increasing utilization rates for compute platforms through virtualization while expanding connectivity options through the deployment of secure remote access. The architecture demonstrates the maturity of the technologies generally available in industry today and the ability to successfully abstract the tightly coupled relationship between thick-client software and legacy hardware into a hardware agnostic "Infrastructure as a Service" capability that can scale to meet future requirements of new space programs and spacecraft. This paper discusses the benefits and difficulties that a migration to cloud-based computing philosophies has uncovered when compared to the legacy Mission Control Center architecture. The team consists of system and software engineers with extensive experience with the MCC infrastructure and software currently used to support the International Space Station (ISS) and Space Shuttle program (SSP).
Martin, Keith↗
Hydrological Data at the NASA GES DISC: Current Capabilities and New Opportunities
The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is one of twelve NASA Earth science data centers that document, process, archive and distribute data from Earth observation missions and projects. GES DISC maintains an archive of several hydrology datasets, including the Land Data Assimilation Systems (LDAS) and the Gravity Recovery and Climate Experiment (GRACE) Data Assimilation for Drought Monitoring (GRACE-DA-DM) data products. These datasets include model output of heat fluxes, rain, snow, soil temperature, soil moisture, and runoff; and observational forcing data, including surface pressure, temperature, precipitation, downward shortwave and longwave radiation, humidity, and wind. The temporal resolution of the hydrology data at GES DISC ranges from hourly to monthly, and spatial resolutions range from 0.1° to 1.0°. The GES DISC provides services which enable users to aggregate, temporally and spatially subset, regrid, and visualize archived data including the GES DISC Subsetter, Hydrology Data Rods, and the Geospatial Interactive Online Visualization and Analysis Infrastructure (GIOVANNI). The Hydrology Data Rods service optimally reorganizes large hydrological data sets as extended time series, providing more efficient access for the hydrological community. The time series data (aka “data rods”) were integrated into hydrology community tools, such as the Data Rods Explorer on HydroShare. Furthermore, the GES DISC is in the process of migrating its data and services to the cloud. Hydrological data available at the GES DISC are now available in the Amazon Web Services (AWS) cloud (us-west-2 region) providing users Direct S3 data access and the capability for cloud computing operations. In this presentation, the hydrology data products and services currently available at the GES DISC will be summarized. Also discussed are the migration to the cloud, user support through this transition, and the status of migrating the data rods service to the cloud.
Ashley Heath↗
NASA's Implementation of Cloud Services for Human Space Flight
Cloud is a tried-and-true technology used throughout United States government agencies, including the National Aeronautics and Space Administration (NASA). With reliable results and infrequent downtimes, cloud allows for secure remote access, customizability, and streamlined monitoring options, creating an environment for better data integrity and availability. As NASA increasingly migrates functions to the cloud, the Space Communications and Navigation Program (SCaN) program has been investigating how this capability can be leveraged to provide communication services to its users and customers. Currently, missions such as NASA-ISRO Synthetic Aperture Radar (NISAR), Plankton, Aerosol, Cloud, ocean Ecosystem (PACE), and Roman Space Telescope (RST) are planned to incorporate cloud into their data delivery architecture. However, SCaN is looking to expand further. This conversion to using cloud services allows for greater availability of mission data for both robotic and human space flight (HSF)missions. The SCaN program and the Near Space Network (NSN) are working to consolidate resources and create a cloud environment suitable for the entirety of the SCaN program network architecture. SCaN is in the process of finalizing its cloud architecture and soon will be implementing cloud services. The new services used will adhere to federal regulations including Federal Risk and Authorization Management Program (FedRAMP), which is built upon National Institute of Standards and Technology (NIST)documentation. While keeping in mind these security requirements, an auxiliary objective of the cloud integration is to ensure the most cost-efficient solution; providing a scalable, robust and resilient system. Using cloud services, NASA will gain access to better centralized monitoring and management features, along with customizable services on a pay-per-use plan. With the ever-growing NASA mission data volume needs, maintaining ample storage space is another major constraint. Processing and storing such large amounts of data, on the order of terabytes a day, requires dynamic processing capability which is inherently a strength of cloud computing. By routing this data from ground stations through the cloud, there will be greater ease of access for both SCaN and the user community. Artificial intelligence and other built-in cloud functions can also enhance efficiency, improving data processing time. Thereby also allowing for better data availability. As we look to the future of cloud services, NASA will continue to leverage capabilities that will benefit NASA’s ability to provide cost-effective communication services. This paper further outlines the evolution of cloud use by SCaN in the context of Human Space Flight.
cloud storage↗
Technical Challenges and Lessons from the Migration of the GLOBE Data and Information System to Utilize Cloud Computing Service
The Global Learning and Observation to Benefit the Environment (GLOBE) Data and Information System supports an international science and education program with capabilities to accept local environment observations, archive, display and visualize them along with global satellite observations. Since its inception twenty years ago, the Web and database system has been upgraded periodically to accommodate the changes in technology and the steady growth of GLOBEs education community and collection of observations. Recently, near the end-of-life of the system hardware, new commercial computer platform options were explored and a decision made to utilize Cloud services. Now the GLOBE DIS has been fully deployed and maintained using Amazon Cloud services for over two years now. This paper reviews the early risks, actual challenges, and some unexpected findings as a result of the GLOBE DIS migration. We describe the plans, cost drivers and estimates, highlight adjustments that were made and suggest improvements. We present the trade studies for provisioning, for load balancing, networks, processing, storage, as well as production, staging and backup systems. We outline the migration teams skills and required level of effort for transition, and resulting changes in the overall maintenance and operations activities. Examples include incremental adjustments to processing capacity and frequency of backups, and efforts previously expended on hardware maintenance that were refocused onto application-specific enhancements.
GLOBE↗
Technical Challenges and Lessons from the Migration of the GLOBE Data and Information System to Utilize Cloud Computing Service
The Global Learning and Observation to Benefit the Environment (GLOBE) Data and Information System supports an international science and education program with capabilities to accept local environment observations, archive, display and visualize them along with global satellite observations. Since its inception twenty years ago, the Web and database system has been upgraded periodically to accommodate the changes in technology and the steady growth of GLOBEs education community and collection of observations. Recently, near the end-of-life of the system hardware, new commercial computer platform options were explored and a decision made to utilize Cloud services. Now the GLOBE DIS has been fully deployed and maintained using Amazon Cloud services for over two years now. This paper reviews the early risks, actual challenges, and some unexpected findings as a result of the GLOBE DIS migration. We describe the plans, cost drivers and estimates, highlight adjustments that were made and suggest improvements. We present the trade studies for provisioning, for load balancing, networks, processing, storage, as well as production, staging and backup systems. We outline the migration teams skills and required level of effort for transition, and resulting changes in the overall maintenance and operations activities. Examples include incremental adjustments to processing capacity and frequency of backups, and efforts previously expended on hardware maintenance that were refocused onto application-specific enhancements.
GLOBE Data↗
Earthdata Cloud Analytics Project
This presentation describes a nascent project in NASA to develop a framework to support end-user analytics of NASA's Earth science data in the cloud. The chief benefit of migrating EOSDIS (Earth Observation System Data and Information Systems) data to the cloud is to position the data next to enormous computing capacity to allow end users to process data at scale. The Earthdata Cloud Analytics project will user a service-based approach to facilitate the infusion of evolving analytics technology and the integration with non-NASA analytics or other complementary functionality at other agencies and in other nations.
data services↗
Predicting runtime and resource utilization of jobs on integrated cloud and HPC systems
Recent advances in virtualization technologies used in cloud computing offer performance that closely approaches bare-metal levels. Combined with specialized instance types and high-speed networking services for cluster computing, cloud platforms have become a compelling option for high-performance computing (HPC). However, most current batch job schedulers in HPC systems are designed for homogeneous clusters and make decisions based on limited information about jobs and system status. Scientists typically submit computational jobs to these schedulers with a requested runtime that is often over- or under-estimated. More accurate runtime predictions can help schedulers make better decisions and reduce job turnaround times. Here, they can also support decisions about migrating jobs to the cloud to avoid long queue wait times in HPC systems.
97 MATHEMATICS AND COMPUTING↗
CIRA: Cooperative Institute for Research in the Atmosphere Newsletter, Volume 28, Fall 2007
The articles in this issue of the Cooperative Institute for Research in the Atmosphere (CIRA) Newsletter are: "Unmanned Aerial Systems: An Overview of NOAA's Unmanned Aircraft System Program," "International Activities: Weather Briefings and Training Via the Internet," "Cloudsat's One-Year Anniversary: An Abundance of Exciting New Cloud Observations," and "The Migration of NCAR'S Auto-Nowcaster into NWS AWIPS."
McInnis-Efaw, Mary↗
Trade Study: Storing NASA HDF5/netCDF-4 Data in the Amazon Cloud and Retrieving Data via Hyrax Server / THREDDS Data Server
As part of the overall effort to understand implications of migrating ESDIS data and services to the cloud we are testing several common OPeNDAP and HDF use cases against three architectures for general performance and cost characteristics. The architectures include retrieving entire files, retrieving datasets using HTTP range gets, and retrieving elements of datasets (chunks) with HTTP range gets. We will describe these architectures and discuss our approach to estimating cost.
HDF↗
Recent Trends of the Tropical Hydrological Cycle Inferred from Global Precipitation Climatology Project and International Satellite Cloud Climatology Project data
Scores of modeling studies have shown that increasing greenhouse gases in the atmosphere impact the global hydrologic cycle; however, disagreements on regional scales are large, and thus the simulated trends of such impacts, even for regions as large as the tropics, remain uncertain. The present investigation attempts to examine such trends in the observations using satellite data products comprising Global Precipitation Climatology Project precipitation and International Satellite Cloud Climatology Project cloud and radiation. Specifically, evolving trends of the tropical hydrological cycle over the last 20-30 years were identified and analyzed. The results show (1) intensification of tropical precipitation in the rising regions of the Walker and Hadley circulations and weakening over the sinking regions of the associated overturning circulation; (2) poleward shift of the subtropical dry zones (up to 2deg/decade in June-July-August (JJA) in the Northern Hemisphere and 0.3-0.7deg/decade in June-July-August and September-October-November in the Southern Hemisphere) consistent with an overall broadening of the Hadley circulation; and (3) significant poleward migration (0.9-1.7deg/decade) of cloud boundaries of Hadley cell and plausible narrowing of the high cloudiness in the Intertropical Convergence Zone region in some seasons. These results support findings of some of the previous studies that showed strengthening of the tropical hydrological cycle and expansion of the Hadley cell that are potentially related to the recent global warming trends.
Zhou, Y. P.↗
ncompare: A Python Package for Comparing netCDF Structures
Earth science researchers and data engineers have a common problem: they often need to compare data files to see what is different between them. A lot of time is spent developing code to test differences. When it comes to comparing multidimensional data file formats like netCDFs (Network Common Data Form), this is particularly challenging and time-consuming, since there is frequently a need to evaluate the differences between dimension sizes, variable structures, and variable attributes, especially for regression testing. Since netCDFs are widely used in Earth science — with climate models, oceanographic or atmospheric reanalyses, and observational data — improved means of evaluating netCDF files can help enable a wide range of applications. We have developed a reusable open source approach through `ncompare`, which is a Python package for comparing netCDF structures [[https://github.com/nasa/ncompare]]. The `ncompare` tool compares the structure of two Network Common Data Form (NetCDF) files at the command line. It facilitates rapid comparisons by generating a formatted display of the matching and non-matching groups, variables, and associated metadata between two NetCDF datasets. The user has the option to colorize the terminal output for ease of viewing, and `ncompare` can optionally save comparison reports in text, comma-separated value (CSV), and/or Microsoft Excel formats. Despite the availability of tools (such as ncmpidiff or nccmp) that compare the values of variables, there was not previously a readily available, Python-based tool for rapid visual comparisons of group and variable structures, attributes, and chunking. `ncompare` was developed at NASA’s Atmospheric Science Data Center (ASDC) and is a collaboration with NASA Openscapes [[https://nasa-openscapes.github.io]] mentors across 11 of NASA’s data centers. Openscapes’ overarching vision is to support scientific researchers using NASA Earthdata as they migrate their workflows to the cloud. Relevant links: - https://github.com/nasa/ncompare - https://github.com/pyOpenSci/software-submission/issues/146 - https://nasa-openscapes.github.io
Daniel Kaufman↗
Regulation of Electron Cloud Density by Electronic Effects of Substituents to Optimize Photocatalytic H 2 Evolution of Carbon Dots
Carbon dots (CDs) have a wide light absorption range, which are promising candidates for photocatalytic water splitting H 2 evolution, but they show low activity for hydrogen evolution reaction (HER) due to the slow transfer efficiencies of photogenerated carriers. The electron cloud density of photocatalyst is essential for the separation and migration of photogenerated carriers. In this study, the effect mechanism of electron cloud density regulated by the substituents with different electronic effects on the photocatalytic HER of glucose-based CDs is clarified. CDs-SO 3 H and CDs-OH are first obtained by introducing electron-drawing group (–SO 3 H) and electron-donating group (–OH) using a tailoring post-processing strategy. Experimental results show that the H 2 yield catalyzed by CDs-SO 3 H is 89.95 µmol•g –1 in 4 h, which is about four times that of CDs, and 7.4 times that of CDs-OH. The –SO 3 H induces a much negative energy band and high electron cloud density on CDs edges, promoting the separation and transfer of photogenerated carriers; while the CDs-OH exhibits a high positive charge density and an upward energy band, hindering the surface complexation of electron-hole pairs and HER. Furthermore, this study will provide an insight into the design of CDs catalysts with efficient photocatalytic HER at a molecular level.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
What is LAADS DAAC? How to Download Data from NASA's Source for Cloud and Aerosol Data
The Level-1 and Atmosphere Archive Distribution System (LAADS) Distributed Active Archive Center (DAAC) primarily archives and distributes data on clouds and aerosols in Earth’s atmosphere in addition to providing Level 0 and 1 Moderate Imaging Radiometer Suite (MODIS), Visible Infrared Imaging Radiometer Suite (VIIRS) and airborne data as well as data from the European Space Agency’s Sea and Land Surface Temperature Radiometer (SLSTR) and the Ocean and Land Colour Instrument (OLCI). Secondarily, LAADS DAAC serves as a backup source for MODIS and VIIRS land products. As one of twelve DAACS supported by NASA’s Earth Science Data and Information System (ESDIS), LAADS DAAC provides a gateway to data through multiple tools. The newest, View Data, allows users to preview data sets before downloading or applying any post-processing transformations. Learn about how to access LAADS DAAC data, get up-to-date information about LAADS DAAC's migration to hosting data in the cloud, and discover how to use the View Data tool.
Tassia Owen↗
Cloud Surprises in Moving NASA EOSDIS Applications into Amazon Web Services
NASA ESDIS has been moving a variety of data ingest, distribution, and science data processing applications into a cloud environment over the last 2 years. As expected, there have been a number of challenges in migrating primarily on-premises applications into a cloud-based environment, related to architecture and taking advantage of cloud-based services. What was not expected is a number of issues that were beyond purely technical application re-architectures. We ran into surprising network policy limitations, billing challenges in a government-based cost model, and difficulty in obtaining certificates in an NASA security-compliant manner. On the other hand, this approach has allowed us to move a number of applications from local hosting to the cloud in a matter of hours (yes, hours!!), and our CMR application now services 95% of granule searches and an astonishing 99% of all collection searches in under a second. And most surprising of all, well, you'll just have to wait and see the realization that caught our entire team off guard!
Cloud↗
Using OPeNDAP In The Cloud to Connect NASA Data Centers and Support Open Data Access
NASA DAACs (Distributed Active Archive Centers), including the Goddard Earth Sciences Data Information and Services Center (GES DISC), are currently transitioning from on-premises servers to a shared Earthdata Cloud in order to build more interoperability, cross-collaboration, and streamlined services between their data centers. Migrating its on-premises OPeNDAP service to the cloud is a critical component of making this interconnectedness between NASA DAACs possible. To improve their cloud services, GES DISC is leveraging open-source platforms like Github to collect user feedback, create use cases, and develop resources to enable real-time learning for users about OPeNDAP in the cloud. This presentation gives an overview of the OPeNDAP in the cloud, resources developed to access this service, and considerations for improved user guidance and experience to further support NASA's commitment to the Open-Source Science Initiative (OSSI).
Christopher Battisto↗