SEARCH · Engineering Papers

Results for “data requests”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

All sources NASA NTRS DOE OSTI

NASA NTRS2003

A Distributed Data Architecture for 2001 Mars Odyssey Data Distribution

Newer instruments and communications techniques have given scientists unprecedented amounts of data, more than can be feasibly distributed through traditional methods such as mailed CD-ROM's. Leveraging the web makes sense since it enables scientists to request specific data and retrieve products as soon as they're available. Yet defining the middleware system to support such an application has remained just out of reach, until Odyssey. For the first time ever, data from all Odyssey mission instruments were made available through a single system immediately upon delivery to the Planetary Data System (PDS). The Object Oriented Data Technology (OODT) software made such an application possible.

distributed↗

NASA NTRS2005

Step 1: C3 Flight Demo Data Analysis Plan

The Data Analysis Plan (DAP) describes the data analysis that the C3 Work Package (WP) will perform in support of the Access 5 Step 1 C3 flight demonstration objectives as well as the processes that will be used by the Flight IPT to gather and distribute the data collected to satisfy those objectives. In addition to C3 requirements, this document will encompass some Human Systems Interface (HSI) requirements in performing the C3 flight demonstrations. The C3 DAP will be used as the primary interface requirements document between the C3 Work Package and Flight Test organizations (Flight IPT and Non-Access 5 Flight Programs). In addition to providing data requirements for Access 5 flight test (piggyback technology demonstration flights, dedicated C3 technology demonstration flights, and Airspace Operations Demonstration flights), the C3 DAP will be used to request flight data from Non- Access 5 flight programs for C3 related data products

Source record↗

NASA NTRS1989

The worldwide ionospheric data base

The worldwide ionospheric data base is scattered over the entire globe. Different data sets are held at different institutions in the U.S., U.S.S.R., Australia, Europe, and Asia. The World Data Centers on the different continents archive and distribute part of the huge data base; the scope and cross section of the individual data holdings depend on the regional and special interest of the center. An attempt is made to pull together all the strings that point toward different ionospheric data holdings. Requesters are provided with the information about what is available and where to get it. An attempt is also made to evaluate the reliability and compatibility of the different data sets based on the consensus in the ionospheric research community. The status and accuracy of the standard ionospheric models are also discussed because they may facilitate first order assessment of ionospheric effects. This is a first step toward an ionospheric data directory within the framework of NSSDC's master directory.

Bilitza, Dieter↗

NASA NTRS2010

Autonomous Sensorweb Operations for Integrated Space, In-Situ Monitoring of Volcanic Activity

We have deployed and demonstrated operations of an integrated space in-situ sensorweb for monitoring volcanic activity. This sensorweb includes a network of ground sensors deployed to the Mount Saint Helens volcano as well as the Earth Observing One spacecraft. The ground operations and space operations are interlinked in that ground-based intelligent event detections can cause the space segment to acquire additional data via observation requests and space-based data acquisitions (thermal imagery) can trigger reconfigurations of the ground network to allocate increased bandwidth to areas of the network best situated to observe the activity. The space-based operations are enabled by an automated mission planning and tasking capability which utilizes several Opengeospatial Consortium (OGC) Sensorweb Enablement (SWE) standards which enable acquiring data, alerts, and tasking using web services. The ground-based segment also supports similar protocols to enable seamless tasking and data delivery. The space-based segment also supports onboard development of data products (thermal summary images indicating areas of activity, quicklook context images, and thermal activity alerts). These onboard developed products have reduced data volume (compared to the complete images) which enables them to be transmitted to the ground more rapidly in engineering channels.

Chien, Steve A.↗

NASA NTRS2021

HAPI: An API Standard for Accessing Heliophysics Time Series Data

Heliophysics data analysis often involves combining diverse science measurements, many of them captured as time series. Although there are now only a few commonly used data file formats, the diversity in mechanisms for automated access to and aggregation of such data holdings can make analysis that requires intercomparison of data from multiple data providers difficult. The Heliophysics Application Programmer's Interface (HAPI) is a recently developed standard for accessing distributed time series data to increase interoperability. The HAPI specification is based on the common elements of existing data services, and it standardizes the two main parts of a data service: the request interface and the response data structures. The interface is based on the REpresentational State Transfer (REST) or RESTful architecture style, and the HAPI specification defines five required REST endpoints. Data are returned via a streaming format that hides file boundaries; the metadata is detailed enough for the content to be scientifically useful, e.g., plotted with appropriate axes layout, units, and labels. Multiple mature HAPI-related open-source projects offer server-side implementation tools and client-side libraries for reading HAPI data in multiple languages (IDL, Java, MATLAB, and Python). Multiple data providers in the US and Europe have added HAPI access alongside their existing interfaces. Based on this experience, data can be served via HAPI with little or no information loss compared to similar existing web interfaces. Finally, HAPI has been recommended as a COSPAR standard for time series data delivery.

Robert S. Weigel↗

NASA NTRS2015

Evolution of Archival Storage (from Tape to Memory)

Over the last three decades, there has been a significant evolution in storage technologies supporting archival of remote sensing data. This section provides a brief survey of how these technologies have evolved. Three main technologies are considered - tape, hard disk and solid state disk. Their historical evolution is traced, summarizing how reductions in cost have helped being able to store larger volumes of data on faster media. The cost per GB of media is only one of the considerations in determining the best approach to archival storage. Active archives generally require faster response to user requests for data than permanent archives. The archive costs have to consider facilities and other capital costs, operations costs, software licenses, utilities costs, etc. For meeting requirements in any organization, typically a mix of technologies is needed.

Magnetic Storage; Technology Evolution↗

DOE OSTI2025

Hanford 200 West Area Flowsheet Data to Support Waste Treatment and Disposal Request for Proposal

This report documents the 200 West Area (200W) flowsheet supplemental data needed to support the Request for Proposal (RFP) to procure onsite and/or offsite treatment and disposal capabilities for West Area pretreated1 tank waste (PTW). The data provided in this document is based on the results of a 200W flowsheet model run evaluating single-shell tank (SST) retrievals for all S, SX, and U Farms, except for Tank S-112, which was retrieved in March of 2007 (HNF-EP-0182, Waste Tank Summary Report for Month End August 31, 2024). The evaluation also includes the waste inventory in double-shell tanks (DSTs) in SY Farm.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2017

Accessing Data Federations with CVMFS

Data federations have become an increasingly common tool for large collaborations such as CMS and Atlas to efficiently distribute large data files. Unfortunately, these typically are implemented with weak namespace semantics and a non-POSIX API. On the other hand, CVMFS has provided a POSIX-compliant read-only interface for use cases with a small working set size (such as software distribution). The metadata required for the CVMFS POSIX interface is distributed through a caching hierarchy, allowing it to scale to the level of about a hundred thousand hosts. In this paper, we will describe our contributions to CVMFS that merges the data scalability of XRootD-based data federations (such as AAA) with metadata scalability and POSIX interface of CVMFS. We modified CVMFS so it can serve unmodified files without copying them to the repository server. CVMFS 2.2.0 is also able to redirect requests for data files to servers outside of the CVMFS content distribution network. Finally, we added the ability to manage authorization and authentication using security credentials such as X509 proxy certificates. We combined these modifications with the OSGs StashCache regional XRootD caching infrastructure to create a cached data distribution network. Here, we will show performance metrics accessing the data federation through CVMFS compared to direct data federation access. Additionally, we will discuss the improved user experience of providing access to a data federation through a POSIX filesystem.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NASA NTRS2004

Overview of NASA's Earth Science Data Systems

For over the last 15 years, NASA's Earth Science Enterprise (ESE) has devoted a tremendous effort to design and build the Earth Observing System (EOS) Data and Information System (EOSDIS) to acquire, process, archive and distribute the data of the EOS series of satellites and other ESE missions and field programs. The development of EOSDIS began with an early prototype to support NASA data from heritage missions and progressed through a formal development process to today's system that supports the data from multiple missions including Landsat 7, Terra, Aqua, SORCE and ICESat. The system is deployed at multiple Distributed Active Archive Centers (DAACs) and its current holdings are approximately 4.5 petabytes. The current set of unique users requesting EOS data and information products exceeds 2 million. While EOSDIS has been the centerpiece of NASA's Earth Science Data Systems, other initiatives have augmented the services of EOSDIS and have impacted its evolution and the future directions of data systems within the ESE. ESDIS had an active prototyping effort and has continued to be involved in the activities of the Earth Science Technology Office (ESTO). In response to concerns from the science community that EOSDIS was too large and monolithic, the ESE initiated the Earth Science Information Partners (ESP) Federation Experiment that funded a series of projects to develop specialized products and services to support Earth science research and applications. Last year, the enterprise made 41 awards to successful proposals to the Research, Education and Applications Solutions Network (REASON) Cooperative Agreement Notice to continue and extend the ESP activity. The ESE has also sponsored a formulation activity called the Strategy for the Evolution of ESE Data Systems (SEEDS) to develop approaches and decision support processes for the management of the collection of data system and service providers of the enterprise. Throughout the development of its earth science data systems, NASA has had an active collaboration with a number of interagency and international partners. One of the mechanisms that has been extremely helpful in initiating and promoting this collaboration has been NASA's participation in the Committee on Earth Observation Satellites (CEOS) and its Working Group on Information Systems and Services (WGISS). The CEOS members, working together, have implemented an International Directory Network that enables users to locate collections of earth science data held by the international community and an International Catalog System to search and order specific data products. CEOS WGISS has also promoted the international interest in the Open GIS Consortium s specifications that further advance the access and use of geospatial data and the interoperation of GTS components. These are just a few highlights of the benefits that member agencies gain from CEOS participation.

McDonald, Kenneth↗

NASA NTRS2016

What's NEW at the GES DISC: Evolution of Data Management and Services for Aura Mission and Beyond

GES DISC world. Aura data usage and trend. Aura data users requests. GES DISC update (before/after); New Access method (ftp to http) with Earthdata Login System, New Website (DISC/Mirador to New Interface), New Giovanni (Giovanni to Now Federated). GES DISC support beyond Aura Mission; Multi-sensor coincident data subsets, Level 2 support (Sub-setter, Visualization), Data List.

metrics↗

DOE OSTI2022

Latency hiding for caches

A technique for accessing a memory having a high latency portion and a low latency portion is provided. The technique includes detecting a promotion trigger to promote data from the high latency portion to the low latency portion, in response to the promotion trigger, copying cache lines associated with the promotion trigger from the high latency portion to the low latency portion, and in response to a read request, providing data from either or both of the high latency portion or the low latency portion, based on a state associated with data in the high latency portion and the low latency portion.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2011

Integration and Cooperation in the Next Golden Age of Human Space Flight Data Repositories: Tools for Retrospective Analysis and Future Planning

As NASA transitions from the Space Shuttle era into the next phase of space exploration, the need to ensure the capture, analysis, and application of its research and medical data is of greater urgency than at any other previous time. In this era of limited resources and challenging schedules, the Human Research Program (HRP) based at NASA s Johnson Space Center (JSC) recognizes the need to extract the greatest possible amount of information from the data already captured, as well as focus current and future research funding on addressing the HRP goal to provide human health and performance countermeasures, knowledge, technologies, and tools to enable safe, reliable, and productive human space exploration. To this end, the Science Management Office and the Medical Informatics and Health Care Systems Branch within the HRP and the Space Medicine Division have been working to make both research data and clinical data more accessible to the user community. The Life Sciences Data Archive (LSDA), the research repository housing data and information regarding the physiologic effects of microgravity, and the Lifetime Surveillance of Astronaut Health Repository (LSAH-R), the clinical repository housing astronaut data, have joined forces to achieve this goal. The task of both repositories is to acquire, preserve, and distribute data and information both within the NASA community and to the science community at large. This is accomplished via the LSDA s public website (http://lsda.jsc.nasa.gov), which allows access to experiment descriptions including hardware, datasets, key personnel, mission descriptions and a mechanism for researchers to request additional data, research and clinical, that is not accessible from the public website. This will result in making the work of NASA and its partners available to the wider sciences community, both domestic and international. The desired outcome is the use of these data for knowledge discovery, retrospective analysis, and planning of future research studies.

Thomas, D.↗

NASA NTRS2011

NASA Life Sciences Data Repositories: Tools for Retrospective Analysis and Future Planning

As NASA transitions from the Space Shuttle era into the next phase of space exploration, the need to ensure the capture, analysis, and application of its research and medical data is of greater urgency than at any other previous time. In this era of limited resources and challenging schedules, the Human Research Program (HRP) based at NASA s Johnson Space Center (JSC) recognizes the need to extract the greatest possible amount of information from the data already captured, as well as focus current and future research funding on addressing the HRP goal to provide human health and performance countermeasures, knowledge, technologies, and tools to enable safe, reliable, and productive human space exploration. To this end, the Science Management Office and the Medical Informatics and Health Care Systems Branch within the HRP and the Space Medicine Division have been working to make both research data and clinical data more accessible to the user community. The Life Sciences Data Archive (LSDA), the research repository housing data and information regarding the physiologic effects of microgravity, and the Lifetime Surveillance of Astronaut Health (LSAH-R), the clinical repository housing astronaut data, have joined forces to achieve this goal. The task of both repositories is to acquire, preserve, and distribute data and information both within the NASA community and to the science community at large. This is accomplished via the LSDA s public website (http://lsda.jsc.nasa.gov), which allows access to experiment descriptions including hardware, datasets, key personnel, mission descriptions and a mechanism for researchers to request additional data, research and clinical, that is not accessible from the public website. This will result in making the work of NASA and its partners available to the wider sciences community, both domestic and international. The desired outcome is the use of these data for knowledge discovery, retrospective analysis, and planning of future research studies.

Thomas, D.↗

DOE OSTI2020

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

DOE OSTI2020

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific ap- plications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Net- works (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate band- widths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer perfor- mance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application- level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher pre- diction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

DOE OSTI2019

TAZeR: Hiding the Cost of Remote I/O in Distributed Scientific Workflows

A perennial bottleneck in distributed workflow analytics is long access latencies for remote data. We ask the question: assuming that data must be accessed remotely, can latencies be hidden? We present TAZeR, a framework that reduces data access latency while increasing data reuse. TAZeR transparently converts POSIX I/O into operations that interleave application work with data transfer, i.e. read, prefetching, and write stage-out. TAZeR ensures read data moves directly to application memory without synchronous intervention (soft zero-copy). TAZeR uses distributed bandwidth-aware staging to exploit reuse across application tasks and to manage the capacity constraints of fast hierarchical storage. We evaluate TAZeR on a High Energy Physics workflow that requests remote data at 48 Gb/s (over two 1 Gb/s WAN links) using complex access patterns. TAZeR is 12× and 22× faster than XRootD (state-of-the-art) and file copies (current approach), respectively; and within 7% of optimal. We discuss conditions when TAZeR can hide I/O accesses; and evaluate performance as effective staging sizes change.

Tazer, remote I/O, data staging↗

DOE OSTI2020

On Performance Prediction of Big Data Transfer in High-performance Networks

Big data generated by large-scale scientific and industrial applications need to be transferred between different geographical locations for remote storage, processing, and analysis. High-speed dedicated connections provisioned in High-performance Networks (HPNs) are increasingly utilized to carry out such big data transfer. HPN management highly relies on an important capability of performance (mainly throughput) prediction to reserve sufficient bandwidth and meanwhile avoid over-provisioning that may result in unnecessary resource waste. This capability is critical to improving the resource (mainly bandwidth) utilization of dedicated connections and meeting various user requests for data transfer. Conventional methods conduct performance prediction by fitting prior observed transfer history with predefined loss functions, without considering unobservable latent factors such as competing loads on end hosts. Such latent factors also have a significant impact on the application-level data transfer performance, which may result in an inaccurate prediction model. In this paper, we first investigate the impact of latent factors and propose a clustering-based method to eliminate their negative impact on performance prediction. We then develop a robust machine learning-based performance predictor by: i) incorporating the proposed latent factor elimination method into data preprocessing, and ii) adopting a customized domain guided loss function. Extensive experimental results show that our predictor achieves significantly higher prediction accuracy than several other state-of-the-art methods.

Liu, Wuji↗

NASA NTRS2019

The Seismic Experiment for Interior Structure (SEIS): Experiment Data Distribution

The six sensors of SEIS (The Seismic Experiment for Interior Structure) [- one of three primary instruments on NASA's Mars Lander Insight] cover a broad range of the seismic bandwidth, from 0.01 hertz to 50 hertz, with possible extension to longer periods. Data are transmitted in the form of three continuous VBB (Very Broad-Band) components at 2 samples per second (sps), an estimation of the short period (SP) energy content from the SP at 1 sps, and a continuous compound VBB/SP vertical axis at 10 sps. The continuous streams are augmented by requested event data with sample rates from 20 to 100 sps. SEIS data products are downlinked from the spacecraft in raw CCSDS (Consultative Committee for Space Data Systems) packets and converted to both the Standard for the Exchange of Earthquake Data (SEED) format files and ASCII tables (GeoCSV) for analysis and archiving. Metadata are available in dataless SEED and StionXML. Time series data (waveforms) are available in miniseed and GeoCSV. Data are distributed according to FDSN (Federation of Digital Seismograph Networks - http://www.fdsn.org) formats and interfaces. Wind, pressure and temperature data from the Auxiliary Payload Sensor Suite (APSS) will also be available in SEED format, and can be used for decorrelation and diagnostic purposes on SEIS.

Pardo, Constanza↗