Engineering PapersSearch

SEARCH · Engineering Papers

Results for “data center modeling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Open-Source Data Engineering at NASA: CCMC's Approach to Managing Petabyte-Scale Heliophysics Data

The Community Coordinated Modeling Center (CCMC) at NASA Goddard Space Flight Center (GSFC) leads heliophysics research by providing open access to numerous models and their outputs. Our resources are available on-demand and continuously updated with real-time data, covering sun-earth interactions across multiple domains. These domains include coronal, heliosphere, inner and global magnetosphere, ionosphere, thermosphere, and lower atmosphere interactions. Operating in a hybrid environment, CCMC utilizes both self-owned hardware and Amazon Web Services (AWS) cloud infrastructure. Managing petabytes of data across multiple locations necessitates robust data engineering solutions. To address this challenge, CCMC has adopted industry-standard and open-source tools. We use Apache Airflow as our primary data engineering platform, Python for scripting and data processing, and GitLab for version control and CI/CD. Additionally, we employ Kubernetes for containerized services, Grafana and Prometheus for metrics and monitoring, and Terraform and Puppet for reproducible infrastructure as code. This presentation will discuss lessons learned from our data engineering experiences, platforms evaluated but found unsuitable for our scientific data requirements, and specific techniques developed to enhance data transfer speed and reliability. By using these technologies effectively, CCMC continues to advance heliophysics research through efficient data management and open-access modeling.

space weather

Development of a Web-Based Visualization Platform for Climate Research Using Google Earth

Recently, it has become easier to access climate data from satellites, ground measurements, and models from various data centers, However, searching. accessing, and prc(essing heterogeneous data from different sources are very tim -consuming tasks. There is lack of a comprehensive visual platform to acquire distributed and heterogeneous scientific data and to render processed images from a single accessing point for climate studies. This paper. documents the design and implementation of a Web-based visual, interoperable, and scalable platform that is able to access climatological fields from models, satellites, and ground stations from a number of data sources using Google Earth (GE) as a common graphical interface. The development is based on the TCP/IP protocol and various data sharing open sources, such as OPeNDAP, GDS, Web Processing Service (WPS), and Web Mapping Service (WMS). The visualization capability of integrating various measurements into cE extends dramatically the awareness and visibility of scientific results. Using embedded geographic information in the GE, the designed system improves our understanding of the relationships of different elements in a four dimensional domain. The system enables easy and convenient synergistic research on a virtual platform for professionals and the general public, gr$tly advancing global data sharing and scientific research collaboration.

Sun, Xiaojuan

Overview of NASA MSFC and UAH Space Weather Modeling and Data Efforts

Marshall Space Flight Center, along with its industry and academia neighbors, has a long history of space environment model development and testing. Space weather efforts include research, testing, model development, environment definition, anomaly investigation, and operational support. This presentation will highlight a few of the current space weather activities being performed at Marshall and through collaborative efforts with University of Alabama in Huntsville scientists.

Parker, Linda Neergaard

Statistical Analysis of Model Data for Operational Space Launch Weather Support at Kennedy Space Center and Cape Canaveral Air Force Station

The 12-km resolution North American Mesoscale (NAM) model (MesoNAM) is used by the 45th Weather Squadron (45 WS) Launch Weather Officers at Kennedy Space Center (KSC) and Cape Canaveral Air Force Station (CCAFS) to support space launch weather operations. The 45 WS tasked the Applied Meteorology Unit to conduct an objective statistics-based analysis of MesoNAM output compared to wind tower mesonet observations and then develop a an operational tool to display the results. The National Centers for Environmental Prediction began running the current version of the MesoNAM in mid-August 2006. The period of record for the dataset was 1 September 2006 - 31 January 2010. The AMU evaluated MesoNAM hourly forecasts from 0 to 84 hours based on model initialization times of 00, 06, 12 and 18 UTC. The MesoNAM forecast winds, temperature and dew point were compared to the observed values of these parameters from the sensors in the KSC/CCAFS wind tower network. The data sets were stratified by model initialization time, month and onshore/offshore flow for each wind tower. Statistics computed included bias (mean difference), standard deviation of the bias, root mean square error (RMSE) and a hypothesis test for bias = O. Twelve wind towers located in close proximity to key launch complexes were used for the statistical analysis with the sensors on the towers positioned at varying heights to include 6 ft, 30 ft, 54 ft, 60 ft, 90 ft, 162 ft, 204 ft and 230 ft depending on the launch vehicle and associated weather launch commit criteria being evaluated. These twelve wind towers support activities for the Space Shuttle (launch and landing), Delta IV, Atlas V and Falcon 9 launch vehicles. For all twelve towers, the results indicate a diurnal signal in the bias of temperature (T) and weaker but discernable diurnal signal in the bias of dewpoint temperature (T(sub d)) in the MesoNAM forecasts. Also, the standard deviation of the bias and RMSE of T, T(sub d), wind speed and wind direction indicated the model error increased with the forecast period all four parameters. The hypothesis testing uses statistics to determine the probability that a given hypothesis is true. The goal of using the hypothesis test was to determine if the model bias of any of the parameters assessed throughout the model forecast period was statistically zero. For th is dataset, if this test produced a value >= -1 .96 or <= 1.96 for a data point, then the bias at that point was effectively zero and the model forecast for that point was considered to have no error. A graphical user interface (GUI) was developed so the 45 WS would have an operational tool at their disposal that would be easy to navigate among the multiple stratifications of information to include tower locations, month, model initialization times, sensor heights and onshore/offshore flow. The AMU developed the GUI using HyperText Markup Language (HTML) so the tool could be used in most popular web browsers with computers running different operating systems such as Microsoft Windows and Linux.

Bauman, William H., III

Federated Giovanni: A Distributed Web Service for Analysis and Visualization of Remote Sensing Data

The Geospatial Interactive Online Visualization and Analysis Interface (Giovanni) is a popular tool for users of the Goddard Earth Sciences Data and Information Services Center (GES DISC) and has been in use for over a decade. It provides a wide variety of algorithms and visualizations to explore large remote sensing datasets without having to download the data and without having to write readers and visualizers for it. Giovanni is now being extended to enable its capabilities at other data centers within the Earth Observing System Data and Information System (EOSDIS). This Federated Giovanni will allow four other data centers to add and maintain their data within Giovanni on behalf of their user community. Those data centers are the Physical Oceanography Distributed Active Archive Center (PO.DAAC), MODIS Adaptive Processing System (MODAPS), Ocean Biology Processing Group (OBPG), and Land Processes Distributed Active Archive Center (LP DAAC). Three tiers are supported: Tier 1 (GES DISC-hosted) gives the remote data center a data management interface to add and maintain data, which are provided through the Giovanni instance at the GES DISC. Tier 2 packages Giovanni up as a virtual machine for distribution to and deployment by the other data centers. Data variables are shared among data centers by sharing documents from the Solr database that underpins Giovanni's data management capabilities. However, each data center maintains their own instance of Giovanni, exposing the variables of most interest to their user community. Tier 3 is a Shared Source model, in which the data centers cooperate to extend the infrastructure by contributing source code.

Giovanni

EGIS - An Environmental GIS Developed for NASA Field Center Applications

As the principal center for Environmental Geographic Information Systems (EGIS), the John C. Stennis Space Center (SSC), located in Hancock County, Mississippi, has been assigned technical support requirements to design and to implement a basic EGIS data base for all NASA Field Centers. The intent of this Phase I effort is to produce a baseline EGIS data base incorporating newly available remotely sensed data as well as existing environmental data. A example application of the use of the data base at Stennis Space Center will be to illustrate baseline environmental conditions for consideration with proposed propulsion test stand development and operation. To effectively answer questions related to environmental issues at each center, organization of the data layers and sources will include the following categories: Cadastral/Geodetic; Geopolitical; Hydrography; Infrastructure; Physical Geography; Socioeconomic; Remote Sensing Imagery; Associated Metadata. As part of a Phase II effort, site-specific data and applications will be implemented and added to the data base at each Field Center. This poster illustrates the framework of the design and implementation of a basic EGIS data base. Shown are example data sources, hardware and software, and data base delivery and installation. The poster also depicts future recommendations for a centrally located server to house each of the NASA Field Center data bases. The server will allow real-time data base updates with additional layers and models for each center. Expansion of the EGIS data base will continue to grow as site-specific applications are developed addressing the ongoing evolution of environmental concerns at all NASA Field Centers.

Smoot, James

A Unified Level of Service Model for NASA Earth Science Data Stewardship

During the past year, the Interagency Implementation and Concepts Team (IMPACT) reviewed existing service models in use at various NASA data centers in an effort to produce a unified, cohesive, and comprehensive Level of Service Model for all of NASA Distributed Active Archive Centers (DAACs). NASA DAACs are responsible for ensuring NASA Earth Science data are accurately and securely ingested, distributed, supported, and preserved. The term “Service" as used here refers to the spectrum of data management activities and outputs provided by DAACs in support of the data cared fo by each data cente. The unified Level-of- Service (LoS) model described in this presentation utilizes both the NASA-defined data product category and the data processing level to easily identify an appropriate level-of-service to be applied to a data product throughout the full data life cycle. This LoS model is to be used by DAACs when appraising incoming data in order to determine the appropriate and required services to provide. The LoS model utilizes a 3-level system in which services build upon previous levels and thereby require greater commitment and effort both on the part of the DAAC personnel and the data producer at the highest level. The LoS model description also contains examples of ways to communicate with data producers and data users what services can be expected, thereby bringing more consistent user experiences across the enterprise. In this presentation, we will outline the features of the LoS model and describe how it relates to the FAIR data practices and the NOAA Maturity Matrix model.

Smith, Deborah

2008 GEM Modeling Challenge: Metrics Study of the Dst Index in Physics-Based Magnetosphere and Ring Current Models and in Statistical and Analytic Specifications

In this paper the metrics-based results of the Dst part of the 2008-2009 GEM Metrics Challenge are reported. The Metrics Challenge asked modelers to submit results for 4 geomagnetic storm events and 5 different types of observations that can be modeled by statistical or climatological or physics-based (e.g. MHD) models of the magnetosphere-ionosphere system. We present the results of over 25 model settings that were run at the Community Coordinated Modeling Center (CCMC) and at the institutions of various modelers for these events. To measure the performance of each of the models against the observations we use comparisons of one-hour averaged model data with the Dst index issued by the World Data Center for Geomagnetism, Kyoto, Japan, and direct comparison of one-minute model data with the one-minute Dst index calculated by the United States Geologic Survey (USGS).

Rastaetter, L.

Estimating Rain Attenuation In Satellite Communication Links

Attenuation computed with help of statistical model and meteorological data. NASA Lewis Research Center Satellite Link Attenuation Model (SLAM) program QuickBASIC computer program evaluating static and dynamic statistical assessment of impact of rain attenuation on communication link established between Earth terminal and geosynchronous satellite. Application in specification, design, and assessment of satellite communication links for any terminal location in continental United States. Written in Microsoft QuickBASIC.

Manning, R. M.

Effect of lightning on the concentration of odd nitrogen species in the lower stratosphere: An update

Revised model estimates of the effect of lightning on the lower stratospheric NOy are presented. Several changes in the Atmospheric and Environmental Research Incorporated model were made since the last evaluation of the impact of lightning. Improvements were made in the model circulation and location of tropopause in the tropics, which is now calculated from the National Meteorological Center temperature data. Changes in model circulation reduced the mass flux from the troposphere to the tropical stratosphere. The calculated mass fluxes are found to agree better with some recent estimates. The circulation changes also reduced the advective mass flux from the tropical lower stratosphere to the midlatitudes. The change in circulation and the change in the tropopause height lead to increases in the calculated concentration of N2O, O3 and NOy in the tropical lower stratosphere. The effect of lightning is to increase the calculated concentration of NOy around 64 mbar by a factor of 2, compared to a factor of 10 enhancement in the previous calculations. Comparison with the Stratosphere Troposphere Exchange Project 1987 data indicates that the inclusion of a lightning source brings the model results in closer agreement with the observations.

Kotamarthi, V. R.

Data Format Standardization of Space Weather Model Output at the Community Coordinated Modeling Center

The disparate nature of space weather model output provides many challenges with regards to the portability and reuse of not only the data itself, but also any tools that are developed for analysis and visualization. We are developing and implementing a comprehensive data format standardization methodology that allows heterogeneous model output data to be stored uniformly in any common science data format. We will discuss our approach to identifying core meta-data elements that can be used to supplement raw model output data, thus creating self-descriptive files. The meta-data should also contain information describing the simulation grid. This will ultimately assists in the development of efficient data access tools capable of extracting data at any given point and time. We will also discuss our experiences standardizing the output of two global magnetospheric models, and how we plan to apply similar procedures when standardizing the output of the solar, heliospheric, and ionospheric models that are also currently hosted at the Community Coordinated Modeling Center.

Maddox, M.

Multi-Scale Hydrometeorological Modeling, Land Data Assimilation and Parameter Estimation with the Land Information System

The Land Information System (LIS; http://lis.gsfc.nasa.gov) is a flexible land surface modeling framework that has been developed with the goal of integrating satellite-and ground-based observational data products and advanced land surface modeling techniques to produce optimal fields of land surface states and fluxes. As such, LIS represents a step towards the next generation land component of an integrated Earth system model. In recognition of LIS object-oriented software design, use and impact in the land surface and hydrometeorological modeling community, the LIS software was selected as a co-winner of NASA?s 2005 Software of the Year award.LIS facilitates the integration of observations from Earth-observing systems and predictions and forecasts from Earth System and Earth science models into the decision-making processes of partnering agency and national organizations. Due to its flexible software design, LIS can serve both as a Problem Solving Environment (PSE) for hydrologic research to enable accurate global water and energy cycle predictions, and as a Decision Support System (DSS) to generate useful information for application areas including disaster management, water resources management, agricultural management, numerical weather prediction, air quality and military mobility assessment. LIS has e volved from two earlier efforts -- North American Land Data Assimilation System (NLDAS) and Global Land Data Assimilation System (GLDAS) that focused primarily on improving numerical weather prediction skills by improving the characterization of the land surface conditions. Both of GLDAS and NLDAS now use specific configurations of the LIS software in their current implementations.In addition, LIS was recently transitioned into operations at the US Air Force Weather Agency (AFWA) to ultimately replace their Agricultural Meteorology (AGRMET) system, and is also used routinely by NOAA's National Centers for Environmental Prediction (NCEP)/Environmental Modeling Center (EMC) for their land data assimilation systems to support weather and climate modeling. LIS not only consolidates the capabilities of these two systems, but also enables a much larger variety of configurations with respect to horizontal spatial resolution, input datasets and choice of land surface model through "plugins". LIS has been coupled to the Weather Research and Forecasting (WRF) model to support studies of land-atmosphere coupling be enabling ensembles of land surface states to be tested against multiple representations of the atmospheric boundary layer. LIS has also been demonstrated for parameter estimation, who showed that the use of sequential remotely sensed soil moisture products can be used to derive soil hydraulic and texture properties given a sufficient dynamic range in the soil moisture retrievals and accurate precipitation inputs.LIS has also recently been demonstrated for multi-model data assimilation using an Ensemble Kalman Filter for sequential assimilation of soil moisture, snow, and temperature.Ongoing work has demonstrated the value of bias correction as part of the filter, and also that of joint calibration and assimilation.Examples and case studies demonstrating the capabilities and impacts of LIS for hydrometeorological modeling, assimilation and parameter estimation will be presented as advancements towards the next generation of integrated observation and modeling systems

Peters-Lidard, Christa D.

Modern Era Retrospective Restrospective-Analysis for Research and Applications (MERRA) Data and Services at the GES DISC

The Modern Era Retrospective-analysis for Research and Applications (MERRA) dataset is a NASA satellite era, 30 year (1979 - present), reanalysis using the Goddard Earth Observing System Data Assimilation System, Version 5 (GEOS-5). The project, run out of NASA's Global Modeling and Assimilation Office at Goddard Space Flight Center, provides the science and application communities with a state-of-the-art global analysis with emphasis on improved estimates of the hydrological cycle over a broad range of weather and climate time scales. MERRA products are generated as a long-term synthesis that places the NASA EOS suite of observations in a climate context. The MERRA analysis is performed at a horizontal resolution of 2/3 longitude x 1/2 latitude (540x361 global gridpoints) with observational analyses every 6 hours. The MERRA output data will include 3 dimensional state fields for every 6 hourly analysis cycle on 42 pressure levels (or 72 terrain following model coordinate levels) from the surface through the stratosphere. Several data products are specifically designed to support chemistry and stratosphere transport modeling. The 2 dimensional surface and atmospheric diagnostics (numbering 259) are being stored on the native grid at 1 hourly intervals. These include radiation and vertical integrals of the atmosphere for water and energy budget studies and also surface diagnostics where the diurnal cycle is important. The one hourly surface and near surface data product will also facilitate research on the integrated analysis of Earth system observations in the land, ocean and cryosphere. The MERRA products are archived and distributed by the Goddard Earth Sciences Data and Information Services Center (GES DISC) through its Modeling DISC Web (MDISC) portal. Multiple data access methods and services are available for MERRA data through MDISC: (1) Mirador offers a quick, comprehensive search of MERRA and all GES DISC archived data holdings, allowing searches on keywords, location names or latitude/longitude box, and date/time, with responses within a few seconds. (2) Giovanni is a GES DISC developed Web application that provides data visualization and analysis online. Giovanni features popular visualizations such as latitude-longitude maps, animations, cross sections, profiles, time series, etc. and some basic statistical analysis functions such as scatter plots and correlation coefficient maps. Users are able to download results in several different formats, including Google Earth. (3) On-the-fly parameter subsetting of data within a spatial/temporal window is provided through a simple select and click Web page. (4) MERRA data are also available via OPeNDAP, GrADS Data Server (GDS) and can be converted to netCDF on the fly.

Berrick, Stephen W.

Multi-Scale Hydrometeorological Modeling, Land Data Assimilation and Parameter Estimation with the Land Information System

The Land Information System (LIS; http://lis.gsfc.nasa.gov; Kumar et al., 2006; Peters- Lidard et al.,2007) is a flexible land surface modeling framework that has been developed with the goal of integrating satellite- and ground-based observational data products and advanced land surface modeling techniques to produce optimal fields of land surface states and fluxes. As such, LIS represents a step towards the next generation land component of an integrated Earth system model. In recognition of LIS object-oriented software design, use and impact in the land surface and hydrometeorological modeling community, the LIS software was selected ase co-winner of NASA's 2005 Software of the Year award. LIS facilitates the integration of observations from Earth-observing systems and predictions and forecasts from Earth System and Earth science models into the decision-making processes of partnering agency and national organizations. Due to its flexible software design, LIS can serve both as a Problem Solving Environment (PSE) for hydrologic research to enable accurate global water and energy cycle predictions, and as a Decision Support System (DSS) to generate useful information for application areas including disaster management, water resources management, agricultural management, numerical weather prediction, air quality and military mobility assessment. LIS has evolved from two earlier efforts North American Land Data Assimilation System (NLDAS; Mitchell et al. 2004) and Global Land Data Assimilation System (GLDAS; Rodell al. 2004) that focused primarily on improving numerical weather prediction skills by improving the characterization of the land surface conditions. Both of GLDAS and NLDAS now use specific configurations of the LIS software in their current implementations. In addition, LIS was recently transitioned into operations at the US Air Force Weather Agency (AFWA) to ultimately replace their Agricultural Meteorology (AGRMET) system, and is also used routinely by NOAA's National Centers for Environmental Prediction (NCEP)/Environmental Modeling Center (EMC) for their land data assimilation systems to support weather and climate modeling. LIS not only consolidates the capabilities of these two systems, but also enables a much larger variety of configurations with respect to horizontal spatial resolution, input datasets and choice of land surface model through "plugins,". As described in Kumar et al., 2007, and demonstrated in Case et al., 2008, and Santanello et al., 2009, LIS has been coupled to the Weather Research and Forecasting (WRF) model to support studies of land-atmosphere coupling the enabling ensembles of land surface states to be tested against multiple representations of the atmospheric boundary layer. LIS has also been demonstrated for parameter estimation as described in Peters-Lidard et al. (2008) and Santanello et al. (2007), who showed that the use of sequential remotely sensed soil moisture products can be used to derive soil hydraulic and texture properties given a sufficient dynamic range in the soil moisture retrievals and accurate precipitation inputs. LIS has also recently been demonstrated for multi-model data assimilation (Kumar et al., 2008) using an Ensemble Kalman Filter for sequential assimilation of soil moisture, snow, and temperature. Ongoing work has demonstrated the value of bias correction as part of the filter, and also that of joint calibration and assimilation. Examples and case studies demonstrating the capabilities and impacts of LIS for hydrometeoroogical modeling, assimilation and parameter estimation will be presented as advancements towards the next generation of integrated observation and modeling systems.

Peters-Lidard, Christa D.

A Technique Using Principal Component Analysis to Compare Seasonal Cycles of Earth Radiation from CERES and Model Computations

A method for quantitatively comparing the seasonal cycles of two global data sets is presented. The seasonal cycles of absorbed solar radiation (ASR) and outgoing longwave radiation (OLR) have been computed from an eight-year data set from the Clouds and Earth's Radiant Energy System (CERES) scanning radiometers and from a model data set produced by the NASA Goddard Space Flight Center Global Modeling and Assimilation Office. To compare the seasonal cycles from these two data sets, principal component (PC) analysis is used, where the PCs express the time variations and the corresponding empirical orthogonal functions (EOFs) describe the geographic variations. Ocean has a long thermal response time compared to land, so land and ocean are separated for the analysis. The root-mean square values for the seasonal cycles of ASR and OLR are extremely close for the two data sets. The first three PCs are quite close, showing that the time responses and magnitudes over the globe are very similar. The agreement between the two sets of PCs is quantified by computing the matrix of inner products of the two sets. For ASR over land, the first PCs of CERES and the model agree to better than 99.9%. The EOF maps are similar for most of the globe, but differ in a few places, and the agreement of the EOF maps is likewise quantified. Maps of differences between the annual cycles show regions of agreement and disagreement.

Smith, G. Louis

Big Data Challenges at CCMC

Like other research centers, the Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) at NASA Goddard Space Flight Center (GSFC) is also experiencing the big data challenges. CCMC hosts over 80 space weather models for Runs On Request (ROR), Continuous Runs and Instant Runs simulation services for the research community. In addition, CCMC has started to support simulation output onboarding in response to the Open Science initiative. Overall, we have accumulated over petabytes of simulation output data and are rapidly growing. In this presentation, we will discuss our data and storage challenges. We will present our attempts to address our challenges and any associated lessons learned. CCMC uses Apache Airflow to ensure data transfer is consistent. We will give a brief overview on how we leverage Apache Airflow to enhance our environment.

space weather

Enhancing Dataset Discovery and Usage Tracking in Earth Sciences: Integrating Knowledge Graphs and Large Language Models

NASA's Data Active Archive Centers (DAACs) have played a crucial role in supporting a wide range of applied research in Earth and Environmental sciences. To date, over 20,000 publications have been collected, citing more than 3,000 NASA Earth science datasets. We present an innovative approach that links datasets and collected publications through a knowledge graph (KG). This KG enables the tracking of dataset citations throughout the dataset's lifecycle, revealing patterns of dataset usage across various applied research areas. We fine-tuned the pre-trained NASA IMPACT INDUS-Base Retriever Large Language Model (LLM) using a set of labeled publication abstracts. Our results indicate that 87% of the publications were classified into one of twenty applied research areas, while the remaining 13% were categorized into non-applied research areas. The classified publications linked to datasets are used to discover datasets by users interested in specific applied research and by dataset providers to determine dataset usage for applications.

open-source