Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “service migration”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

An Approach to Reasoning Service Migration in Data and Reasoning Fabric (DRF) Implementation

In this paper we consider service migration problem for Data and Reasoning Fabric (DRF) enabled airspace operations assuming a fixed cloud/edge infrastructure with allocated computing, storage and power resources, where cloud/edge servers, and communication stations are in a wired connected network, while vehicles use a wireless network for communication. The objective is to automatically select the best location for the requested service execution, which achieves minimum cost while satisfying the user quality of service (QoS) and available resources constraints. To this end, estimates of the response time, consumed energy and total cost are defined for each potential compute location. A mixed-integer linear program is then formulated and solved to identify optimal compute locations given QoS constraints, network infrastructure limitations, with worst-case vehicle positioning. The approach is applied to trajectory re-planning use case to avoid a collision with an emergency vehicle in real time.

Air mobility↗

An Approach to Reasoning Service Migration in Data and Reasoning Fabric (DRF) Implementation

In this paper we consider service placement problem for Data and Reasoning Fabric (DRF) enabled airspace operations assuming a fixed cloud/edge infrastructure with allocated computing, storage and power resources, where cloud/edge servers, and communication stations are in a wired connected network, while vehicles use a wireless network for communication. The objective is to automatically select the best location for the requested service execution, which achieves minimum cost while satisfying the user quality of service (QoS) and available resources constraints. To this end, we estimate for each potential location the response time, consumed energy and total cost; formulate an optimization problem for cost minimization given the users QoS constraints and network infrastructure limitations; and solve it using nonlinear programming tools. The approach is applied to trajectory re-planning use case to handle a no-fly zone contingency in real time.

Vahram Stepanyan↗

DMFS: A Data Migration File System for NetBSD

I have recently developed dmfs, a Data Migration File System, for NetBSD. This file system is based on the overlay file system, which is discussed in a separate paper, and provides kernel support for the data migration system being developed by my research group here at NASA/Ames. The file system utilizes an underlying file store to provide the file backing, and coordinates user and system access to the files. It stores its internal meta data in a flat file, which resides on a separate file system. Our data migration system provides archiving and file migration services. System utilities scan the dmfs file system for recently modified files, and archive them to two separate tape stores. Once a file has been doubly archived, files larger than a specified size will be truncated to that size, potentially freeing up large amounts of the underlying file store. Some sites will choose to retain none of the file (deleting its contents entirely from the file system) while others may choose to retain a portion, for instance a preamble describing the remainder of the file. The dmfs layer coordinates access to the file, retaining user-perceived access and modification times, file size, and restricting access to partially migrated files to the portion actually resident. When a user process attempts to read from the non-resident portion of a file, it is blocked and the dmfs layer sends a request to a system daemon to restore the file. As more of the file becomes resident, the user process is permitted to begin accessing the now-resident portions of the file. For simplicity, our data migration system divides a file into two portions, a resident portion followed by an optional non-resident portion. Also, a file is in one of three states: fully resident, fully resident and archived, and (partially) non-resident and archived. For a file which is only partially resident, any attempt to write or truncate the file, or to read a non-resident portion, will trigger a file restoration. Truncations and writes are blocked until the file is fully restored so that a restoration which only partially succeed does not leave the file in an indeterminate state with portions existing only on tape and other portions only in the disk file system. We chose layered file system technology as it permits us to focus on the data migration functionality, and permits end system administrators to choose the underlying file store technology. We chose the overlay layered file system instead of the null layer for two reasons: first to permit our layer to better preserve meta data integrity and second to prevent even root processes from accessing migrated files. This is achieved as the underlying file store becomes inaccessible once the dmfs layer is mounted. We are quite pleased with how the layered file system has turned out. Of the 45 vnode operations in NetBSD, 20 (forty-four percent) required no intervention by our file layer - they are passed directly to the underlying file store. Of the twenty five we do intercept, nine (such as vop_create()) are intercepted only to ensure meta data integrity. Most of the functionality was concentrated in five operations: vop_read, vop_write, vop_getattr, vop_setattr, and vop_fcntl. The first four are the core operations for controlling access to migrated files and preserving the user experience. vop_fcntl, a call generated for a certain class of fcntl codes, provides the command channel used by privileged user programs to communicate with the dmfs layer.

Studenmund, William↗

NASA’s Global Imagery Browse Services and Worldview Migrate to Mapbox Vector Tiles: The Drawbacks and Advantages

NASA’s Global Imagery Browse Services (GIBS) is in the process of migrating their vector data from raster tiles to Mapbox Vector Tiles (MVTs). MVTs are becoming ubiquitous in the world of web mapping. Serving data in place of pixels for each feature presents numerous new visualization opportunities as well as various new challenges to overcome. This presentation will cover some of the hardships these teams encountered, the techniques used to overcome those barriers, and it will give you a glimpse at how the user experience was enhanced.

Benjamin King↗

FIRE: A Failure-Adaptive RL Framework for Edge Computing Migrations

In edge computing, users' service profiles are migrated between edge servers due to user mobility. Reinforcement Learning (RL) frameworks have been proposed to do so, often trained on simulated data. However, existing RL frameworks overlook occasional server failures, which although rare, impact latency-sensitive applications like AR/VR and real- time obstacle detection. These rare failures, being not adequately represented in historical training data, pose a challenge for data-driven RL algorithms. We introduce FIRE, a framework that adapts to rare events by training a RL policy in an edge computing digital twin environment. We propose FIRE-ImRE, an importance sampling-based Q-learning algorithm, which samples rare events proportionally to their impact on the value function. FIRE considers delay, migration, failure, and backup placement costs across individual and shared service profiles. We prove FIRE-ImRE's boundedness and convergence to optimality. Next, we introduce novel deep Q-learning (FIRE-ImDQL) and actor critic (FIRE-ImACRE) versions of our algorithm to enhance scalability. Here, we extend our framework to accommodate users with varying risk tolerances of rare failure events. Through trace-driven experiments, we show that FIRE reduces edge computing costs compared to vanilla RL and the greedy baseline in the event of failures.

Edge computing↗

Cementitious Barriers Partnership - FY2015 End-Year Report

The DOE-EM Office of Tank Waste Management Cementitious Barriers Partnership (CBP) is chartered with providing the technical basis for implementing cement-based waste forms and radioactive waste containment structures for long-term disposal. Therefore, the CBP ultimate purpose is to support progress in final treatment and disposal of legacy waste and closure of High-Level Waste (HLW) tanks in the DOE complex. This status report highlights the CBP 2015 Software and Experimental Program efforts and accomplishments that support DOE needs in environmental cleanup and waste disposal. DOE needs in this area include: Long-term performance predictions to provide credibility (i.e., a defensible technical basis) for regulator and DOE review and approvals, Facility flow sheet development/enhancements, and Conceptual designs for new disposal facilities. In 2015, the CBP developed a beta release of the CBP Software Toolbox – “Version 3.0”, which includes new STADIUM carbonation and damage models, a new SRNL module for estimating hydraulic properties and flow in fractured and intact cementitious materials, and a new LeachXS/ORCHESTRA (LXO) oxidation module. In addition, the STADIUM sulfate attack and chloride models have been improved as well as the LXO modules for sulfate attack, carbonation, constituent leaching, and percolation with radial diffusion (for leaching and transport in cracked cementitious materials). These STADIUM and LXO models are applicable to and can be used by both DOE and the Nuclear Regulatory Commission (NRC) end-users for service life prediction and long-term leaching evaluations of radioactive waste containment structures across the DOE complex.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Performance assessment for the geological disposal of Deep Burn spent fuel using TTBX

The behavior of Deep Burn Modular High Temperature Reactor Spent Fuel (DBSF) is investigated in the Yucca Mountain geological repository (YMR) with respect to the annual dose (Sv/yr) delivered to the Reasonably Maximally Exposed Individual (RMEI) from the transport of radionuclides released from the graphite waste matrix. Transport calculations are performed with a novel computer code, TTBX which is capable of modeling transport pathways that pass through heterogeneous geological formations. TTBX is a multi-region extension of the existing single region TTB transport code. Overall the peak annual dose received by the RMEI is seen to be four orders of magnitude lower than the regulatory threshold for exposure, even under pessimistic scenarios. A number of factors contribute to the favorable performance of DBSF. A reduction of one order of magnitude in the peak annual dose received by the RMEI is observed for every order of magnitude increase in the waste matrix lifetime, highlighting the importance of the waste matrix durability and suggesting graphite's utility as a potential waste matrix for the disposal of high-level waste. Furthermore, we see that by incorporating a higher fidelity far-field model the peak annual dose calculated to be received by the RMEI is reduced by two orders of magnitude. By accounting for the heterogeneities of the far field we have simultaneously removed unnecessary conservatisms and improved the fidelity of the transport model. (authors)

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Fish Migration, Dams, and Loss of Ecosystem Services in the Mekong Basin

The past decade has seen increased international recognition of the importance of the services provided by natural ecosystems. It is unclear however whether such international awareness will lead to improved environmental management in many regions. We explore this issue by examining the specific case of fish migration and dams on the Mekong river. We determine that dams on the Mekong mainstem and major tributaries will have a major impact on the basin's fisheries and the people who depend upon them for food and income. We find no evidence that current moves towards dam construction will stop, and consider two scenarios for the future of the fisheries and other ecosystems of the basin. We conclude that major investment is required in innovative technology to reduce the loss of ecosystem services, and alternative livelihood strategies to cope with the losses that do occur

13 HYDRO ENERGY↗

A Programmable SDN+NFV Architecture for UAV Telemetry Monitoring

With the explosive growth in UAV numbers forecast worldwide, a core concern is how to manage the ad-hoc network configuration required for mobility management. As UAVs migrate among ground control stations, associated network services, routing and operational control must also rapidly migrate to ensure a seamless transition. In this paper, we present a novel, lightweight and modular architecture which supports high mobility, resilience and flexibility through the application of SDN and NFV principles on top of the UAV infrastructure. By combining SDN programmability and Network Function Virtualization we can achieve resilient infrastructure migration of network services, such as network monitoring and anomaly detection, coupled with migrating UAVs to enable high mobility management. Our container-based monitoring and anomaly detection Network Functions (NFs) can be tuned to specific UAV models providing operators better insight during live, high-mobility deployments. We evaluate our architecture against telemetry from over 80flights from a scientific research UAV infrastructure.

Resilience↗

A message-queuing framework for STAR's online monitoring and metadata collection

We report our experience on migrating STARs Online Services (Run Control System, Data Acquisition System, Slow Control System and Subsystem Monitoring) from direct read/write database accesses to a modern non-blocking message-oriented infrastructure. Based on the Advanced Messaging Queuing Protocol (AMQP) and standards, this novel approach does not specify the message data structure, allowing great flexibility in its use. After careful consideration, we chose Google Protocol Buffers as our primary (de)serialization format for structured data exchange. This migration allows us to reduce the overall system complexity and greatly improve the reliability of the metadata collection and the performance of our online services in general. We will present this new framework through its software architecture overview, providing details about our staged and non-disruptive migration process as well as details of the implementation of pluggable components to provide future improvements without compromising stability and availability of services.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

GENESIS: GPS Environmental and Earth Science Information System

This presentation reviews the GPS ENvironmental and Earth Science Information System (GENESIS). The objectives of GENESIS are outlined (1) Data Archiving, searching and distribution for science data products derived from Space borne TurboRogue Space Receivers for GPS science and other ground based GPS receivers, (2) Data browsing using integrated visualization tools, (3) Interactive web/java-based data search and retrieval, (4) Data subscription service, (5) Data migration from existing GPS archived data, (6) On-line help and documentation, and (7) participation in the WP-ESIP federation. The presentation reviews the products and services of Genesis, and the technology behind the system.

Hajj, George↗

Using OPeNDAP In The Cloud to Connect NASA Data Centers and Support Open Data Access

NASA DAACs (Distributed Active Archive Centers), including the Goddard Earth Sciences Data Information and Services Center (GES DISC), are currently transitioning from on-premises servers to a shared Earthdata Cloud in order to build more interoperability, cross-collaboration, and streamlined services between their data centers. Migrating its on-premises OPeNDAP service to the cloud is a critical component of making this interconnectedness between NASA DAACs possible. To improve their cloud services, GES DISC is leveraging open-source platforms like Github to collect user feedback, create use cases, and develop resources to enable real-time learning for users about OPeNDAP in the cloud. This presentation gives an overview of the OPeNDAP in the cloud, resources developed to access this service, and considerations for improved user guidance and experience to further support NASA's commitment to the Open-Source Science Initiative (OSSI).

Christopher Battisto↗

Hydrological Data at the NASA GES DISC: Current Capabilities and New Opportunities

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is one of twelve NASA Earth science data centers that document, process, archive and distribute data from Earth observation missions and projects. GES DISC maintains an archive of several hydrology datasets, including the Land Data Assimilation Systems (LDAS) and the Gravity Recovery and Climate Experiment (GRACE) Data Assimilation for Drought Monitoring (GRACE-DA-DM) data products. These datasets include model output of heat fluxes, rain, snow, soil temperature, soil moisture, and runoff; and observational forcing data, including surface pressure, temperature, precipitation, downward shortwave and longwave radiation, humidity, and wind. The temporal resolution of the hydrology data at GES DISC ranges from hourly to monthly, and spatial resolutions range from 0.1° to 1.0°. The GES DISC provides services which enable users to aggregate, temporally and spatially subset, regrid, and visualize archived data including the GES DISC Subsetter, Hydrology Data Rods, and the Geospatial Interactive Online Visualization and Analysis Infrastructure (GIOVANNI). The Hydrology Data Rods service optimally reorganizes large hydrological data sets as extended time series, providing more efficient access for the hydrological community. The time series data (aka “data rods”) were integrated into hydrology community tools, such as the Data Rods Explorer on HydroShare. Furthermore, the GES DISC is in the process of migrating its data and services to the cloud. Hydrological data available at the GES DISC are now available in the Amazon Web Services (AWS) cloud (us-west-2 region) providing users Direct S3 data access and the capability for cloud computing operations. In this presentation, the hydrology data products and services currently available at the GES DISC will be summarized. Also discussed are the migration to the cloud, user support through this transition, and the status of migrating the data rods service to the cloud.

Ashley Heath↗

Final Report: Migration Mechanisms for Large-scale Parallel Applications

Process migration is the ability to transfer a process from one machine to another. It is a useful facility in distributed computing environments, especially as computing devices become more pervasive and Internet access becomes more ubiquitous. The potential benefits of process migration, among others, are fault resilience by migrating processes off of faulty hosts, data access locality by migrating processes closer to the data, better system response time by migrating processes closer to users, dynamic load balancing by migrating processes to less loaded hosts, and improved service availability and administration by migrating processes before host maintenance so that applications can continue to run with minimal downtime. Although process migration provides substantial potential benefits and many approaches have been considered, achieving transparent process migration functionality has been difficult in practice. To address this problem, our work has designed, implemented, and evaluated new and powerful transparent process checkpoint-restart and migration mechanisms for desktop, server, and parallel applications that operate across heterogeneous cluster and mobile computing environments. A key aspect of this work has been to introduce lightweight operating system virtualization to provide processes with private, virtual namespaces that decouple and isolate processes from dependencies on the host operating system instance. This decoupling enables processes to be transparently checkpointed and migrated without modifying, recompiling, or relinking applications or the operating system. Building on this lightweight operating system virtualization approach, we have developed novel technologies that enable (1) coordinated, consistent checkpoint-restart and migration of multiple processes, (2) fast checkpointing of process and file system state to enable restart of multiple parallel execution environments and time travel, (3) process migration across heterogeneous software environments, (4) network checkpoint-restart and migration of distributed and parallel applications, (5) a utility computing infrastructure for mobile desktop cloud computing based on process checkpoint-restart and migration functionality, (6) a process migration security architecture for protecting applications and infrastructure from denial-of-service attacks, and (7) a checkpoint-restart mobile computing system using portable storage devices.

99 GENERAL AND MISCELLANEOUS↗

KNBD: A Remote Kernel Block Server for Linux

I am developing a prototype of a Linux remote disk block server whose purpose is to serve as a lower level component of a parallel file system. Parallel file systems are an important component of high performance supercomputers and clusters. Although supercomputer vendors such as SGI and IBM have their own custom solutions, there has been a void and hence a demand for such a system on Beowulf-type PC Clusters. Recently, the Parallel Virtual File System (PVFS) project at Clemson University has begun to address this need (1). Although their system provides much of the functionality of (and indeed was inspired by) the equivalent file systems in the commercial supercomputer market, their system is all in user-space. Migrating their 10 services to the kernel could provide a performance boost, by obviating the need for expensive system calls. Thanks to Pavel Machek, the Linux kernel has provided the network block device (2) with kernels 2.1.101 and later. You can configure this block device to redirect reads and writes to a remote machine's disk. This can be used as a building block for constructing a striped file system across several nodes.

Becker, Jeff↗

Quantifying effectiveness of failure prediction and response in HPC systems : methodology and example.

Effective failure prediction and mitigation strategies in high-performance computing systems could provide huge gains in resilience of tightly coupled large-scale scientific codes. These gains would come from prediction-directed process migration and resource servicing, intelligent resource allocation, and checkpointing driven by failure predictors rather than at regular intervals based on nominal mean time to failure. Given probabilistic associations of outlier behavior in hardware-related metrics with eventual failure in hardware, system software, and/or applications, this paper explores approaches for quantifying the effects of prediction and mitigation strategies and demonstrates these using actual production system data. We describe context-relevant methodologies for determining the accuracy and cost-benefit of predictors. While many research studies have quantified the expected impact of growing system size, and the associated shortened mean time to failure (MTTF), on application performance in large-scale high-performance computing (HPC) platforms, there has been little if any work to quantify the possible gains from predicting system resource failures with significant but imperfect accuracy. This possibly stems from HPC system complexity and the fact that, to date, no one has established any good predictors of failure in these systems. Our work in the OVIS project aims to discover these predictors via a variety of data collection techniques and statistical analysis methods that yield probabilistic predictions. The question then is, 'How good or useful are these predictions?' We investigate methods for answering this question in a general setting, and illustrate them using a specific failure predictor discovered on a production system at Sandia.

99 GENERAL AND MISCELLANEOUS↗

The Future of NASA Earth Science in the Commercial Cloud: Challenges and Opportunities

NASA produces a large volume and variety of data products that are used every day to support research, decision making, and education. The widespread use of NASA’s Earth Science data is enabled by NASA’s Earth Science Data System (ESDS) program, which oversees the archiving and distribution of these data and invests in the development of new data systems and tools. However, NASA’s current approach to Earth Science data distribution — based on distributed institutional archives with individual on-premises high-performance computing capabilities — faces some significant challenges, including massive increases in data volume from upcoming missions, a greater need for transdisciplinary science that synthesizes many different kinds of observations, and a push to make science more open, inclusive, and accessible. To address these challenges, NASA is aggressively migrating its Earth Science data and related tools and services into the commercial cloud. Migration of data into the commercial cloud can significantly improve NASA’s existing data system capabilities by (1) providing more flexible options for storage and compute (including rapid, as-needed access to state-of-the-art capabilities); (2) by centralizing and standardizing data access, which gives all of NASA’s institutional data centers access to all of each other’s datasets; and (3) by facilitating “analysis-in-place”, whereby users can bring their own computational workflows and tools to the data rather than having to maintain their own copies of NASA datasets. However, migration to the commercial cloud also poses some significant challenges, including (1) managing costs under a “pay-as-you-go” model; (2) incompatibility with existing tools and data formats with object-based storage and network access; (3) vendor lock-in; (4) challenges with data access for workflows that mix on-premise and cloud computing; and (5) standardization for highly diverse data as is present in NASA’s data archive. I conclude with two examples of recent NASA activities showcasing capabilities enabled by the commercial cloud: An interactive analysis and development platform for analyzing airborne imaging spectroscopy data, and a new collection of tools and services for data discovery, analysis, publication, and data-driven storytelling (Visualization, Exploration, and Data Analysis, VEDA).

Alexey N Shiklomanov↗