Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “virtual machine”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

Staghorn: An Automated Large-Scale Distributed System Analysis Platform

Conducting experiments on large-scale distributed computing systems is becoming significantly easier with the assistance of emulation. Researchers can now create a model of a distributed computing environment and then generate a virtual, laboratory copy of the entire system composed of potentially thousands of virtual machines, switches, and software. The use of real software, running at clock rate in full virtual machines, allows experiments to produce meaningful results without necessitating a full understanding of all model components. However, the ability to inspect and modify elements within these models is bound by the limitation that such modifications must compete with the model, either running in or alongside it. This inhibits entire classes of analyses from being conducted upon these models. We developed a mechanism to snapshot an entire emulation-based model as it is running. This allows us to \freeze time" and subsequently fork execution, replay execution, modify arbitrary parts of the model, or deeply explore the model. This snapshot includes capturing packets in transit and other input/output state along with the running virtual machines. We were able to build this system in Linux using Open vSwitch and Kernel Virtual Machines on top of Sandia's emulation platform Firewheel. This primitive opens the door to numerous subsequent analyses on models, including state space exploration, debugging distributed systems, performance optimizations, improved training environments, and improved experiment repeatability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Efficient Parallel Discrete Event Simulation on Cloud/Virtual Machine Platforms

Cloud and Virtual machine (VM) technologies present new challenges with respect to performance and monetary cost in executing parallel discrete event simulation (PDES) applications. Due to the introduction of overall cost as a metric, the traditional use of the highest-end computing configuration is no longer the most obvious choice. Moreover, the unique runtime dynamics and configuration choices of Cloud and VM platforms introduce new design considerations and runtime characteristics specific to PDES over Cloud/VMs. Here, an empirical study is presented to guide an understanding of the dynamics, trends, and trade-offs in executing PDES on Cloud/VM platforms. Performance and cost measures obtained from multiple PDES applications executed on the Amazon EC2 Cloud and on a high-end VM host machine reveal new, counterintuitive VM--PDES dynamics and guidelines. One of the critical aspects uncovered is the fundamental mismatch in hypervisor scheduler policies designed for general cloud workloads versus the virtual time ordering needed for PDES workloads. This insight is supported by experimental data revealing the gross deterioration in PDES performance traceable to VM scheduling policy. To overcome this fundamental problem, the design and implementation of a new deadlock-free scheduler algorithm are presented, optimized specifically for PDES applications on VMs. The scalability of our scheduler has been tested up to 128 VMs multiplexed on 32 cores, showing significant improvement in the runtime relative to the default Cloud/VM scheduler. The observations, algorithmic design, and results are timely for emerging cloud/VM-based installations, highlighting the need for PDES-specific support in high performance discrete event simulations on Cloud/VM platforms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Build and Execute Environment

At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.

Guan, Qiang↗
DOE OSTI2017

Build and Execute Environment

At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.

Guan, Qiang↗
DOE OSTI2024

Virtual Synchronous Machine Grid-Forming Inverter Model Specification (REGFM_B1)

This report describes a generic virtual synchronous machine (VSM) grid-forming inverter (GFM) model - REGFM_B1. The initial model specification was proposed by Pacific Northwest National Laboratory (PNNL), General Electric (GE), and Electric Power Research Institute (EPRI). Siemens Gamesa Renewable Energy (SGRE) also provided inputs to the specification. The model specification has been revised multiple times based on the discussions between all the contributors listed in this report. This work was funded by the Universal Interoperability for Grid-Forming Inverters (UNIFI) Consortium. This generic model is developed to help the utility industry understand the concept of VSM GFMs. The model could be used to represent equipment for long-term planning studies where vendor-specific models are not available. As equipment mature and improve, generic models will be updated to capture the new functionalities of GFMs. It is not intended that these models will always remain representative of all future GFM technologies.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2018

Sandia Cloud Forensics and Incident Response Platform v 1.0

As systems and devices become virtualized and deployed in the cloud, the hypervisor becomes an increasingly appropriate place to collect performance data, system state, system landscape, function calls, transaction traces, and other characteristics. We developed a method by which an introspection application may be coupled with a hypervisor to “reach into” the VM with minimal intrusiveness to collect data critical to the reconstruction of events, files, and operations. Such a capability is required to take advantage of the hypervisor as an instrumentation platform and to integrate that data with more traditional collection mechanisms. The concept of a VM serviced by a lightweight hypervisor is a relatively new paradigm for forensic practitioners. Traditional forensic techniques, based on assumptions that the filesystem was directly interacting with the hardware through an abstraction, afforded the forensic practitioner the assumption that there was nothing controlling the application below the filesystem. This is not the case when using virtualized technologies. Hypervisors can covertly monitor, introspect and interact with the guest in a transparent fashion. The problems of storage and collection of actionable data are exhausting. The current challenge is most hypervisors do not expose a useful application programming interface (API) at a sufficient level to do transparent, fine-grained and customizable introspection. Scalable VM instrumentation and introspection at an in-depth level requires fast handling of events, as well as direct access to VM state. Furthermore, deep introspection benefits greatly from the ability to gather data from the hardware during the VM’s exit to the hypervisor. All of this requires identical access to the system as the hypervisor itself; improper use of this ability could easily cause system instability. It is for this reason we believe that the hypervisor developers have been hesitant to grant this much control through their APIs. However, our approach leverages other means to collect and monitor the guests in a targeted fashion. Virtual machine introspection (VMI) is a technique used to monitor the runtime state of a system-level virtual machine. The runtime state can be may include processor registers, memory, disk, network, and any other hardware-level events A review of research literature and current VMI technologies exposed several limitations and tradeoffs in VMI approaches, including: the use of in-guest agents; kernel to user space transitions (dramatically slowing down processing); VMI tool pre-configuration requirements; hypervisor version lock-in or source code patching; reliance on operating system symbols; limited processor features due to hypervisor (even if the hardware could do more). To address these constraints, a VMI tool was envisioned to provide the cloud forensic capabilities while having as few of these limitations as possible. The Kernel-based Virtual Machine Introspection (KVMi) tool was developed for hypervisors on Intel’s x86-64 architecture. To meet performance, scoping and use-case demands, the follow criteria were applied to KVMi to support live, forensic data collection: • Shall not require in-guest agents. • Shall work with any recent version of modern hypervisors. • Shall introspect upon any guest (VM) running modern versions of Windows, Linux and OS X. • Shall have the ability to find and track operating system artifacts • Shall not require operating system symbol files. • Shall be able to fully handle VM-exits, bypassing execution of the hypervisor if necessary, to facilitate new features the hypervisor may not support. • Shall be compilable/loadable on a running system with standard build tools.

Stout, WilliamM.S.↗
DOE OSTI2021

IEEE P2988: Recommended Practice of Use and Functions of Virtual Synchronous Machines

Power systems are going through a paradigm shift from centralized generation to distributed generation. This brings unprecedented challenges to power systems stability, reliability, and security. Operating power electronic converters as Virtual Synchronous Machines (VSMs) can bring backward compatibility to power systems, unify and harmonize the integration and interaction of future power systems players, and improve power system stability, reliability, and security. This is vital for the large-scale adoption of distributed energy resources, the advancement of sustainability, and the development of a low-carbon economy. This IEEE 2988 Recommended Practice will define the fundamental principles, mandatory functions, and optional functions of a VSM, i.e., a power electronic converter that is operated to behave like conventional synchronous machines. It will not describe or specify power semiconductor devices, hardware topologies, or micro-controllers that are used to build a VSM. This talk will discuss the activities and development plan for IEEE 2988 with the aim to engaging broad stakeholders and widening participation.

Zhong, Qing-Chang↗
DOE OSTI2017

Charliecloud

Charliecloud is a set of scripts to let users run a virtual cluster of virtual machines (VMs) on a desktop or supercomputer. Key functions include: 1. Creating (typically by installing an operating system from vendor media) and updating VM images; 2. Running a single VM; 3. Running multiple VMs in a virtual cluster. The virtual machines can talk to one another over the network and (in some cases) the outside world. This is accomplished by calling external programs such as QEMU and the Virtual Distributed Ethernet (VDE) suite. The goal is to let users have a virtual cluster containing nodes where they have privileged access, while isolating that privilege within the virtual cluster so it cannot affect the physical compute resources. Host configuration enforces security; this is not included in Charliecloud, though security guidelines are included in its documentation and Charliecloud is designed to facilitate such configuration. Charliecloud manages passing information from host computers into and out of the virtual machines, such as parameters of the virtual cluster, input data specified by the user, output data from virtual compute jobs, VM console display, and network connections (e.g., SSH or X11). Parameters for the virtual cluster (number of VMs, RAM and disk per VM, etc.) are specified by the user or gathered from the environment (e.g., SLURM environment variables). Example job scripts are included. These include computation examples (such as a "hello world" MPI job) as well as performance tests. They also include a security test script to verify that the virtual cluster is appropriately sandboxed. Tests include: 1. Pinging hosts inside and outside the virtual cluster to explore connectivity; 2. Port scans (again inside and outside) to see what services are available; 3. Sniffing tests to see what traffic is visible to running VMs; 4. IP address spoofing to test network functionality in this case; 5. File access tests to make sure host access permissions are enforced. This test script is not a comprehensive scanner and does not test for specific vulnerabilities. Importantly, no information about physical hosts or network topology is included in this script (or any of Charliecloud); while part of a sensible test, such information is specified by the user when the test is run. That is, one cannot learn anything about the LANL network or computing infrastructure by examining Charliecloud code.

Priedhorsky, Reid↗
DOE OSTI2015

Charliecloud

Charliecloud is a set of scripts to let users run a virtual cluster of virtual machines (VMs) on a desktop or supercomputer. Key functions include: 1. Creating (typically by installing an operating system from vendor media) and updating VM images; 2. Running a single VM; 3. Running multiple VMs in a virtual cluster. The virtual machines can talk to one another over the network and (in some cases) the outside world. This is accomplished by calling external programs such as QEMU and the Virtual Distributed Ethernet (VDE) suite. The goal is to let users have a virtual cluster containing nodes where they have privileged access, while isolating that privilege within the virtual cluster so it cannot affect the physical compute resources. Host configuration enforces security; this is not included in Charliecloud, though security guidelines are included in its documentation and Charliecloud is designed to facilitate such configuration. Charliecloud manages passing information from host computers into and out of the virtual machines, such as parameters of the virtual cluster, input data specified by the user, output data from virtual compute jobs, VM console display, and network connections (e.g., SSH or X11). Parameters for the virtual cluster (number of VMs, RAM and disk per VM, etc.) are specified by the user or gathered from the environment (e.g., SLURM environment variables). Example job scripts are included. These include computation examples (such as a "hello world" MPI job) as well as performance tests. They also include a security test script to verify that the virtual cluster is appropriately sandboxed. Tests include: 1. Pinging hosts inside and outside the virtual cluster to explore connectivity; 2. Port scans (again inside and outside) to see what services are available; 3. Sniffing tests to see what traffic is visible to running VMs; 4. IP address spoofing to test network functionality in this case; 5. File access tests to make sure host access permissions are enforced. This test script is not a comprehensive scanner and does not test for specific vulnerabilities. Importantly, no information about physical hosts or network topology is included in this script (or any of Charliecloud); while part of a sensible test, such information is specified by the user when the test is run. That is, one cannot learn anything about the LANL network or computing infrastructure by examining Charliecloud code.

Priedhorsky, Reid↗
DOE OSTI2017

Enabling High Availability Service with oVirt Virtualization and CephFS

The online computing environment at STAR has generated demand for high availability of services (HAS) and a resilient uptime guarantee. Such services include databases, webservers, and storage systems that user and sub-systems tend to rely on for their critical workflows. Standard deployment of services on bare metal creates a problem if the fundamental hardware fails or loses connectivity. Additionally, the configuration of redundant failover nodes (a secondary Web service for example) requires constant syncing of configuration and content and sometimes manual interaction for switching hardware (DNS name comes to mind). For this project, we will focus on two tools: oVirt and Ceph. oVirt is an OpenSource virtualization management application that enables central management of hardware nodes, storage, and network resources used to deploy and monitor virtual machines. oVirt supports the deployment of a virtual environment for your data center leveraging automatic provisioning, live migration, and the ability to easily scale the number of hypervisors. oVirt enables the use of multiple storage technologies where you can store virtual machines, images, and templates within one or multiple storage systems. STAR's recent efforts focused on deploying a CephFS POSIX compliant distributed storage system, we would enable the ability to couple our Ceph storage with the oVirt virtualization management system. When designing an intricate system for hosting critical services, it is a requirement to circumvent single points of failure. This work will involve the testing and viability of such an approach along with test cases for high availability, live migration, and service on demand.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2016

Identification of Program Signatures from Cloud Computing System Telemetry Data

Malicious cloud computing activity can take many forms, including running unauthorized programs in a virtual environment. Detection of these malicious activities while preserving the privacy of the user is an important research challenge. Prior work has shown the potential viability of using cloud service billing metrics as a mechanism for proxy identification of malicious programs. Previously this novel detection method has been evaluated in a synthetic and isolated computational environment. In this paper we demonstrate the ability of billing metrics to identify programs, in an active cloud computing environment, including multiple virtual machines running on the same hypervisor. The open source cloud computing platform OpenStack, is used for private cloud management at Pacific Northwest National Laboratory. OpenStack provides a billing tool (Ceilometer) to collect system telemetry measurements. We identify four different programs running on four virtual machines under the same cloud user account. Programs were identified with up to 95% accuracy. This accuracy is dependent on the distinctiveness of telemetry measurements for the specific programs we tested. Future work will examine the scalability of this approach for a larger selection of programs to better understand the uniqueness needed to identify a program. Additionally, future work should address the separation of signatures when multiple programs are running on the same virtual machine.

Nichols, Nicole M.↗
DOE OSTI2017

ClockworkVMs

Graphical user interface for use and modification of virtual machine templates using Hashi Corp packer to perform virtual machine creation.

Nielsen, Roy↗
DOE OSTI2017

ClockworkVMs

Graphical user interface for use and modification of virtual machine templates using Hashi Corp packer to perform virtual machine creation.

Nielsen, Roy↗
DOE OSTI2017

Network Hardware Virtualization for Application Provisioning in Core Networks

We present that service providers and vendors are moving toward a network virtualized core, whereby multiple applications would be treated on their own merit in programmable hardware. Such a network would have the advantage of being customized for user requirements and allow provisioning of next generation services that are built specifically to meet user needs. In this article, we articulate the impact of network virtualization on networks that provide customized services and how a provider's business can grow with network virtualization. We outline a decision map that allows mapping of applications with technology that is supported in network-virtualization - oriented equipment. Analogies to the world of virtual machines and generic virtualization show that hardware supporting network virtualization will facilitate new customer needs while optimizing the provider network from the cost and performance perspectives. A key conclusion of the article is that growth would yield sizable revenue when providers plan ahead in terms of supporting network-virtualization-oriented technology in their networks. To be precise, providers have to incorporate into their growth plans network elements capable of new service deployments while protecting network neutrality. Finally, a simulation study validates our NV-induced model.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Threat emulation framework

A method for emulating threats in virtual network computing environment is provided. The method comprises creating a number of virtual machines in the virtual network computing environment. A number of threat actors are emulated, wherein each threat actor comprises a number of threat artifacts that form a sequence of attack steps against the virtual network computing environment. The threat actors are then deployed against the virtual network computing environment. Behavioral data about actions of the threat actors in the virtual network computing environment is collected, as is performance data about the virtual network computing environment in response to the threat actors. The collected behavioral and performance data is then presented to a user via an interface.

Urias, Vincent↗
DOE OSTI2015

Open access to high-level data and analysis tools in the CMS experiment at the LHC

The CMS experiment, in recognition of its commitment to data preservation and open access as well as to education and outreach, has made its first public release of high-level data under the CC0 waiver: up to half of the proton-proton collision data (by volume) at 7 TeV from 2010 in CMS Analysis Object Data format. CMS has prepared, in collaboration with CERN and the other LHC experiments, an open-data web portal based on Invenio. The portal provides access to CMS public data as well as to analysis tools and documentation for the public. The tools include an event display and histogram application that run in the browser. In addition a virtual machine containing a CMS software environment along with XRootD access to the data is available. Within the virtual machine the public can analyse CMS data, example code is provided. As a result, we describe the accompanying tools and documentation and discuss the first experiences of data use.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2023

Emulation automation and model checking

A method of automating emulations is provided. The method comprising collecting publicly available network data over a predefined time interval, wherein the collected network data might comprise structured and unstructured data. Any unstructured data is converted into structured data. The original and converted structured data is stored in a database and compared to known network vulnerabilities. An emulated network is created according to the collected network data and the comparison of the structured data with known vulnerabilities. Virtual machines are created to run on the emulated network. Director programs and guest actor programs are run on the virtual machines, wherein the actor programs imitate real user behavior on the emulated network. The director programs deliver task commands to the guest actor programs to imitate real user behavior. The imitated behavior is presented to a user via an interface.

Urias, Vincent↗
DOE OSTI2014

Overhead-Aware-Best-Fit (OABF) Resource Allocation Algorithm for Minimizing VM Launching Overhead

FermiCloud is a private cloud developed in Fermi National Accelerator Laboratory to provide elastic and on-demand resources for different scientific research experiments. The design goal of the FermiCloud is to automatically allocate resources for different scientific applications so that the QoS required by these applications is met and the operational cost of the FermiCloud is minimized. Our earlier research shows that VM launching overhead has large variations. If such variations are not taken into consideration when making resource allocation decisions, it may lead to poor performance and resource waste. In this paper, we show how we may use an VM launching overhead reference model to minimize VM launching overhead. In particular, we first present a training algorithm that automatically tunes a given refer- ence model to accurately reflect FermiCloud environment. Based on the tuned reference model for virtual machine launching overhead, we develop an overhead-aware-best-fit resource allocation algorithm that decides where and when to allocate resources so that the average virtual machine launching overhead is minimized. The experimental results indicate that the developed overhead-aware-best-fit resource allocation algorithm can significantly improved the VM launching time when large number of VMs are simultaneously launched.

Wu, Hao↗