Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Distribution”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

RAPIDS: Reconciling Availability, Accuracy, and Performance in Managing Geo-Distributed Scientific Data

In modern science, big data plays an increasingly important role. Many scientific applications, such as running simulations on supercomputers or conducting experiments on advanced instruments, produce huge amount of data at unprecedented speed. Analyzing and understanding such big data is the key for scientists to make scientific breakthroughs. However, data might become unavailable for scientists to access when outages or maintenance of the storage system occur, which severely hinders scientific discovery. To improve the data availability, data duplication and erasure coding (EC) are often used. But as the scientific data gets larger, using these two methods can cause considerable storage and network overhead.In this paper, we propose RAPIDS, a hybrid approach that combines the multigrid-based error-bounded lossy compression with erasure coding, to significantly reduce the storage and network overhead required for maintaining high data availability. Our experiments show that RAPIDS reduces the storage overhead by up to 7.5x and network overhead by up to 3x to achieve the same level of availability compared to the regular EC method. We improve RAPIDS by building two models to optimize the fault tolerance configurations and data gathering strategy. We demonstrate that RAPIDS significantly improves performance when running on many CPU cores in parallel or on GPUs.

Wan, Lipeng↗
DOE OSTI2022

rmap: An R package to plot and compare tabular data on customizable maps across scenarios and time

`rmap` is an R package that allows users to easily plot tabular data (CSV or R data frames) on maps without any Geographic Information Systems (GIS) knowledge. Maps produced by `rmap` are `ggplot` objects and thus capitalize on the flexibility and advancements of the `ggplot2` package and all elements of each map are thus fully customizable. Additionally `rmap` automatically detects and produces comparison maps if the data has multiple scenarios or time periods as well as animations for time series data. Advanced users can load their own shapefiles if desired. `rmap` comes with a range of pre-built color palettes but users can also provide any `R` color palette or create their own as needed. Four different legend types are available to highlight different kinds of data distributions. The input spatial data can be both gridded or polygon data. `rmap` is desgined in particular for comparing spatial data across scenarios and time periods and comes preloaded with standard country, state, and basin maps as well as custom maps compatible with the Global Change Analysis Model (GCAM) spatial boundaries. `rmap` has a growing number of users and its products have been used in multiple multisector dynamics publications as well as a required dependency in other R packages such as `rfasst` and `metis`. `rmap's` automatic processing of tabular data using pre-built map selection, difference map calculations, faceting, and animations offers unique functionality which makes it a powerful and yet simple tool for users looking to explore multi-sector, multi-scenario data across space and time.

58 GEOSCIENCES↗
DOE OSTI2020

Nuclear Physics Network Requirements Review Report

The Energy Sciences Network (ESnet) is the Office of Science’s high-performance network user facility, delivering highly reliable data transport capabilities optimized for the requirements of data-intensive science. In essence, ESnet is the circulatory system that enables the U.S. Department of Energy (DOE) science mission by connecting each and every DOE lab and its user facilities. ESnet is funded and stewarded by the Advanced Scientific Computing Research (ASCR) Program and managed and operated by the Scientific Networking Division at Lawrence Berkeley National Laboratory (LBNL). ESnet is widely regarded as a global leader in the research and education networking community. ESnet connects DOE national laboratories, user facilities, and major experiments so scientists can use remote instruments and computing resources as well as share data with collaborators, transfer large data sets, and access distributed data repositories. While ESnet provides network connectivity, it cannot be characterized as an internet service provider as it is specifically built to provide a range of network services that are tailored to meet the unique requirements of DOE’s data-intensive science.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1979

Data System Implications Derived from User Application Requirements for Satellite Data

An investigation of the data system needs as driven by users of space acquired Earth observation data is documented. Two major categories of users, operational and research, are identified. Limiting data acquisition alleviates some of the delays in processing thus improving timeliness of the delivered product. Trade offs occur between timeliness and data distribution costs, and between data storage and reprocessing. The complexity of the data system requirements to apply space data to users' needs is such that no single analysis suffices to design and implement the optimum system. A series of iterations is required with analyses of the salient problems in a general way, followed by a limited implementation of benefit to some users with a continual upgrade in system capacity, functions, and applications served. The resulting most important requirement for the data system is flexibility to accommodate changing requirements as the system is implemented.

Neiers, J.↗
NASA NTRS1987

The structure of separated flow regions occurring near the leading edge of airfoils including transition

All the time and effort was directed toward acquiring, reducing, and analyzing more hot-wire anemometer data. Some static pressure distribution data were also acquired to support the analysis of the velocity profile data. Laser Doppler Velocimetry data were not acquired due to equipment problems. Included were seven combinations of chord Reynolds number, angle of attack, and acoustic forcing using the NACA 663-018 airfoil. This research has as its objective the detailed documentation of the structure and behavior of the transitional separation bubble and the redeveloping boundary layer after reattachment over an airfoil at low Reynolds numbers. The intent is to further the understanding of the complex flow phenomena so that analytic methods for predicting their formulation and development can be improved. These analytic techniques have applications in the design and performance prediction of airfoils operating in the low Reynolds number flight regime.

Mueller, Thomas J.↗
NASA NTRS1990

Extensibility and limitations of FDDI

Recently two standards for Metropolitan Area Networks (MANs), Fiber Distributed Data Interface (FDDI) and Distributed Queue Dual Bus (DQDB), have emerged as the primary competitors for the MAN arena. Great interest exists in building higher speed networks which support large numbers of node and greater distance, and it is not clear what types of protocols are needed for this type of environment. There is some question as to whether or not these MAN standards can be extended to such environments. The extensibility of FDDI to the Gbps range and a long distance environment is investigated. Specification parameters which affect performance are shown and a measure is provided for predicting utilization of FDDI. A comparison of FDDI at 100 Mbps and 1 Gbps is presented. Some specific problems with FDDI are addressed and modifications which improve the viability of FDDI in such high speed networks are investigated.

Game, David↗
NASA NTRS1993

Data compression opportunities in EOSDIS

The Earth Observing System Data and Information System (EOSDIS) is described in terms of its data volume, data rate, and data distribution requirements. Opportunities for data compression in EOSDIS are discussed.

Kobler, Ben↗
NASA NTRS1995

Using and Distributing Spaceflight Data: The Johnson Space Center Life Sciences Data Archive

Life sciences data collected before, during and after spaceflight are valuable and often irreplaceable. The Johnson Space Center Life is hard to find, and much of the data (e.g. Sciences Data Archive has been designed to provide researchers, engineers, managers and educators interactive access to information about and data from human spaceflight experiments. The archive system consists of a Data Acquisition System, Database Management System, CD-ROM Mastering System and Catalog Information System (CIS). The catalog information system is the heart of the archive. The CIS provides detailed experiment descriptions (both written and as QuickTime movies), hardware descriptions, hardware images, documents, and data. An initial evaluation of the archive at a scientific meeting showed that 88% of those who evaluated the catalog want to use the system when completed. The majority of the evaluators found the archive flexible, satisfying and easy to use. We conclude that the data archive effectively provides key life sciences data to interested users.

Cardenas, J. A.↗
NASA NTRS2002

Toward a Virtual Solar Observatory: Starting Before the Petabytes Fall

NASA is currently engaged in the study phase of a modest effort to establish a Virtual Solar Observatory (VSO). The VSO would serve ground- and space-based solar physics data sets from a distributed network of archives through a small number of interfaces to the scientific community. The basis of this approach, as of all planned virtual observatories, is the translation of metadata from the various sources via source-specific dictionaries so the user will not have to distinguish among keyword usages. A single Web interface should give access to all the distributed data. We present the current status of the VSO, its initial scope, and its relation to the European EGSO effort.

Gurman, J. B.↗
NASA NTRS2004

Overview of GPM Missions's Ground Validation Program

An important element of the internationally structured Global Precipitation Measurement (GPM) mission will be its ground validation research program. Within the last year, the initial architecture of this program has taken shape. This talk will describe that architecture, both in terms of the international program and in terms of the separate regional programs of the principle participating space agencies, i.e., ESA, JAXA, and NASA. There are three overriding goals being addressed in the planning of this program; (1) establishing various new, challenging and important scientific research goals vis-a-vis current ground validation programs supporting satellite retrieval of precipitation; (2) designing the program as an international partnership which operates, out of necessity, heterogeneous sites in terms of their respective observational foci and science thrusts, but anneals itself in terms of achieving a few overarching scientific objectives; and (3) developing a well-designed protocol that allows specific sites or site networks, at their choosing, to operate in a 'supersite' mode - defined as the capability to routinely transmit GV information at low latency to GPM's Precipitation Processing System (PPS). (The PPS is being designed as GPM's data information system, a distributed data system with main centers at the Goddard Space Flight Center (GSFC) within NASA, the Earth Observation Research Center (EORC) within JAXA, and a TBD facility to be identified by the ESA s ESTEC facility in Noordwijk.)

Smith, Eric A.↗
NASA NTRS2004

Software architecture for large scale, distributed, data-intensive systems

This paper presents our experience with OODT, a novel software architectual style, and middlware-based implementation for data-intensive systems. To date, OODT has been successfully evaluated in several different science domains including Cancer Research with the National Cancer Institute (NCI), and Planetary Science with NASA's Planetary Data System (PDS).

software architecture↗
NASA NTRS2019

NASA Global Satellite and Model Data Products and Services for Tropical Cyclone Research

The lack of observations over vast tropical oceans is a major challenge for tropical cyclone research. Satellite observations and model reanalysis data play an important role in filling these- gaps. Established in the mid-1980's, the Goddard Earth Sciences Data and Information Services Center (GES DISC), as one of the 12 NASA data centers, archives and distributes data from several Earth science disciplines such as precipitation, atmospheric dynamics, atmospheric composition, hydrology, including well-known NASA satellite missions (e.g. TRMM, GPM) and model assimilation projects (MERRA-2). Acquiring datasets suitable for tropical cyclone research in a large data archive is a challenge for many, especially for those who are not familiar with satellite or model data. Over the years, the GES DISC has developed user-friendly data services. For example, Giovanni is an online visualization and analysis tool, allowing users to visualize and analyze over 2000 satellite- and model-based variables with a Web browser, without downloading data and software. In this chapter, we will describe data and services at the GES DISC with emphasis on tropical cyclone research. We will also present two case studies and discuss future plans.

Liu, Zhong↗