Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Services, SDS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

4 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2013

SDS: A Framework for Scientific Data Services

Large-scale scientific applications typically write their data to parallel file systems with organizations designed to achieve fast write speeds. Analysis tasks frequently read the data in a pattern that is different from the write pattern, and therefore experience poor I/O performance. In this paper, we introduce a prototype framework for bridging the performance gap between write and read stages of data access from parallel file systems. We call this framework Scientific Data Services, or SDS for short. This initial implementation of SDS focuses on reorganizing previously written files into data layouts that benefit read patterns, and transparently directs read calls to the reorganized data. SDS follows a client-server architecture. The SDS Server manages partial or full replicas of reorganized datasets and serves SDS Clients' requests for data. The current version of the SDS client library supports HDF5 programming interface for reading data. The client library intercepts HDF5 calls and transparently redirects them to the reorganized data. The SDS client library also provides a querying interface for reading part of the data based on user-specified selective criteria. We describe the design and implementation of the SDS client-server architecture, and evaluate the response time of the SDS Server and the performance benefits of SDS.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2013

Expediting Scientific Data Analysis with Reorganization of Data

Data producers typically optimize the layout of data files to minimize the write time. In most cases, data analysis tasks read these files in access patterns different from the write patterns causing poor read performance. In this paper, we introduce Scientific Data Services (SDS), a framework for bridging the performance gap between writing and reading scientific data. SDS reorganizes data to match the read patterns of analysis tasks and enables transparent data reads from the reorganized data. We implemented a HDF5 Virtual Object Layer (VOL) plugin to redirect the HDF5 dataset read calls to the reorganized data. To demonstrate the effectiveness of SDS, we applied two parallel data organization techniques: a sort-based organization on a plasma physics data and a transpose-based organization on mass spectrometry imaging data. We also extended the HDF5 data access API to allow selection of data based on their values through a query interface, called SDS Query. We evaluated the execution time in accessing various subsets of data through existing HDF5 Read API and SDS Query. We showed that reading the reorganized data using SDS is up to 55X faster than reading the original data.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2008

The NPOESS Preparatory Project (NPP) Science Data Segment (SDS) Data Depository and Distribution Element (SD3E) System Architecture

The National Polar-orbiting Operational Environmental Satellite System (NPOESS), the U.S. Government's future low-Earth orbiting satellite system, will monitor global weather and environmental conditions. Serving as a risk reduction for NPOESS, the NPOESS Preparatory Project (NPP) will provide remotely sensed atmospheric, land, ocean, ozone, and sounder data that will serve the meteorological and global climate change scientific communities. The National Aeronautics and Space Administration (NASA) NPP Science Data Segment's (SDS) primary role is to independently assess the quality of the NPP science and environmental data records for their ability to support climate research. The SDS is composed of nine elements; an input element that receives data from the operational agencies and acts as a buffer, a calibration analysis element, five elements devoted to measurement based quality assessment, an element used to test algorithmic improvements, and an element that provides overall science direction. Each element requires a set of sensor specific science data products for their evaluation. There are four NPP sensors that will be flown on the NPP observatory. They are the Visible Infrared Imagining Radiometer Suite (VIIRS), the Advanced Technology Microwave Sounder (ATMS), the Cross-Track Infrared Sounder (CrIS), and the Ozone Mapper/Profiler Suite (OMPS). It is estimated that these four sensors combined will make daily data requests for approximately six terabytes of NPP science products from the operational data providers. As a result, issues associated with duplicate data requests, data transfers of large volumes of diverse products, and data transfer failures raised concerns with respect to the network traffic and bandwidth consumption. Therefore, a central data broker system for receiving and buffering data requests and data products for the SDS was developed. The data element for this system is called the SDS Data Depository and Distribution Element (SD3E). It supports science mission data assessment by assuring the timely and validated acquisition and subsequent transfer of the NPP Science Mission data to the SDS Elements and NPP Science Team. The six science elements that interface with the SD3E span across the NASA Goddard Space Flight Center (GSFC), the NASA Jet Propulsion Laboratory (JPL), and the University of Wisconsin. As the primary communication vehicle for the science elements and science team, the SD3E has an interface to the operational data providers: National Environment Satellite, Data, and Information Service (NESDIS) Interface Data Processing System (IDPS) and the National Oceanic Atmospheric Administration's (NOAA) Comprehensive Large Array-data Stewardship system (CLASS) Archive Data System (ADS), that are responsible for product generation and archive and distribution respectively. The SD3E is designed to be a semi-customizable and semi-automated system. This system is designed to provide flexibility and ease of use for the science users in accessing the latest data products by creating a rolling data cache that temporarily stores the products locally before transferring the data to the SDS Measurement based elements for the land, ocean, atmosphere, sounder, and ozone. This paper describes the design and architecture of one of the nine SDS elements, the SD3E, and how this system has provided a mechanism for efficient data exchange, how it has helped in alleviating some of the network traffic and usage, and how it has contributed to reducing operational costs.

Ho, Evelyn L.↗
NASA NTRS2008

Standardizing Interfaces for External Access to Data and Processing for the NASA Ozone Product Evaluation and Test Element (PEATE)

NASA's traditional science data processing systems have focused on specific missions, and providing data access, processing and services to the funded science teams of those specific missions. Recently NASA has been modifying this stance, changing the focus from Missions to Measurements. Where a specific Mission has a discrete beginning and end, the Measurement considers long term data continuity across multiple missions. Total Column Ozone, a critical measurement of atmospheric composition, has been monitored for'decades on a series of Total Ozone Mapping Spectrometer (TOMS) instruments. Some important European missions also monitor ozone, including the Global Ozone Monitoring Experiment (GOME) and SCIAMACHY. With the U.S.IEuropean cooperative launch of the Dutch Ozone Monitoring Instrument (OMI) on NASA Aura satellite, and the GOME-2 instrumental on MetOp, the ozone monitoring record has been further extended. In conjunction with the U.S. Department of Defense (DoD) and the National Oceanic and Atmospheric Administration (NOAA), NASA is now preparing to evaluate data and algorithms for the next generation Ozone Mapping and Profiler Suite (OMPS) which will launch on the National Polar-orbiting Operational Environmental Satellite System (NPOESS) Preparatory Project (NPP) in 2010. NASA is constructing the Science Data Segment (SDS) which is comprised of several elements to evaluate the various NPP data products and algorithms. The NPP SDS Ozone Product Evaluation and Test Element (PEATE) will build on the heritage of the TOMS and OM1 mission based processing systems. The overall measurement based system that will encompass these efforts is the Atmospheric Composition Processing System (ACPS). We have extended the system to include access to publically available data sets from other instruments where feasible, including non-NASA missions as appropriate. The heritage system was largely monolithic providing a very controlled processing flow from data.ingest of satellite data to the ultimate archive of specific operational data products. The ACPS allows more open access with standard protocols including HTTP, SOAPIXML, RSS and various REST incarnations. External entities can be granted access to various modules within the system, including an extended data archive, metadata searching, production planning and processing. Data access is provided with very fine grained access control. It is possible to easily designate certain datasets as being available to the public, or restricted to groups of researchers, or limited strictly to the originator. This can be used, for example, to release one's best validated data to the public, but restrict the "new version" of data processed with a new, unproven algorithm until it is ready. Similarly, the system can provide access to algorithms, both as modifiable source code (where possible) and fully integrated executable Algorithm Plugin Packages (APPs). This enables researchers to download publically released versions of the processing algorithms and easily reproduce the processing remotely, while interacting with the ACPS. The algorithms can be modified allowing better experimentation and rapid improvement. The modified algorithms can be easily integrated back into the production system for large scale bulk processing to evaluate improvements. The system includes complete provenance tracking of algorithms, data and the entire processing environment. The origin of any data or algorithms is recorded and the entire history of the processing chains are stored such that a researcher can understand the entire data flow. Provenance is captured in a form suitable for the system to guarantee scientific reproducability of any data product it distributes even in cases where the physical data products themselves have been deleted due to space constraints. We are currently working on Semantic Web ontologies for representing the various provenance information. A new web site focusing on consolidating informaon about the measurement, processing system, and data access has been established to encourage interaction with the overall scientific community. We will describe the system, its data processing capabilities, and the methods the community can use to interact with the standard interfaces of the system.

Tilmes, Curt A.↗