Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data services”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

DOE SciDAC’s Earth System Grid Center for Enabling Technologies Final Report for University of Southern California Information Sciences Institute

The mission of the Earth System Grid Federation (ESGF) is to provide the worldwide climate-research community with access to the data, information, model codes, analysis tools, and intercomparison capabilities required to make sense of enormous climate data sets. Its specific goals are to (1) provide an easy-to-use and secure web-based data access environment for data sets; (2) add value to individual data sets by presenting them in the context of other data sets and tools for comparative analysis; (3) address the specific requirements of participating organizations with respect to bandwidth, access restrictions, and replication; (4) ensure that the data are readily accessible through the analysis and visualization tools used by the climate research community; and (5) transfer infrastructure advances to other domain areas. For the ESGF, the U.S. Department of Energy’s (DOE’s) Earth System Grid Center for Enabling Technologies (ESG-CET) team has led international development and delivered a production environment for managing and accessing ultra-scale climate data. This production environment includes multiple national and international climate projects (such as the Community Earth System Model and the Coupled Model Intercomparison Project), ocean model data (such as the Parallel Ocean Program), observation data (Atmospheric Radiation Measurement Best Estimate, Carbon Dioxide Information and Analysis Center, Atmospheric Infrared Sounder, etc.), and analysis and visualization tools, all serving a diverse user community. These data holdings and services are distributed across multiple ESG-CET sites (such as ANL, LANL, LBNL/NERSC, LLNL/PCMDI, NCAR, and ORNL) and at unfunded partner sites, such as the Australian National University National Computational Infrastructure, the British Atmospheric Data Centre, the National Oceanic and Atmospheric Administration Geophysical Fluid Dynamics Laboratory, the Max Planck Institute for Meteorology, the German Climate Computing Centre, the National Aeronautics and Space Administration Jet Propulsion Laboratory, and the National Oceanic and Atmospheric Administration. The ESGF software is distinguished from other collaborative knowledge systems in the climate community by its widespread adoption, federation capabilities, and broad developer base. It is the leading source for present climate data holdings, including the most important and largest data sets in the globalclimate community, and—assuming its development continues—we expect it to be the leading source for future climate data holdings as well. Recently, ESG-CET extended its services beyond data-file access and delivery to include more detailed information products (scientific graphics, animations, etc.), secure binary data-access services (based upon the OPeNDAP Data Access Protocol), and server-side analysis. The latter capabilities allow users to request data subsets transformed through commonly used analysis and intercomparison procedures. As we transition from development activities to production and operations, the ESG-CET team is tasked with making data available to all users seeking to understand, process, extract value from, visualize, and/or communicate it to others—this is of course if funding continues at some level. This ongoing effort, though daunting in scope and complexity, would greatly magnify the value of numerical climate model outputs and climate observations for future national and international climate-assessment reports. The ESG-CET team also faces substantial technical challenges due to the rapidly increasing scale of climate simulation and observational data, which will grow, for example, from less than 50 terabytes for the last Intergovernmental Panel on Climate Change (IPCC) assessment to multiple Petabytes for the next IPCC assessment. In a world of exponential technological change and rapidly growing sophistication in climate data analysis, an infrastructure such as ESGF must constantly evolve if it is to remain relevant and useful.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

"PoliMOR: A Policy Engine \"Made-to-Order\" for Automated and Scalable Data Management in Lustre"

Modern supercomputing systems are increasingly reliant on hierarchical, multi-tiered file and storage system architectures due to cost-performance-capacity trade-offs. Within such multi-tiered systems, data management services are required to maintain healthy utilization, performance, and capacity levels. We present PoliMOR, a pragmatic and reliable policy-driven data management framework. PoliMOR is composed of modular, single-purpose agents that gather file system metadata and enforce policies on storage systems. PoliMOR facilitates automated and scalable data management with customizable agents tailored to HPC facility-specific storage systems and policies. Our evaluations demonstrate the scalability and performance of PoliMOR both by its individual agents and as a collective entity. We believe PoliMOR is widely applicable across HPC facilities with large-scale data management challenges and will garner interest from the HPC community, given its flexible and open-source nature.

George, Anjus↗

IS BLOCKCHAIN A SUITABLE TECHNOLOGY FOR ENSURING THE INTEGRITY OF DATA SHARED BY LIGHTING AND OTHER BUILDING SYSTEMS?

Increasing amounts of data are available from lighting and other building systems. In commercial buildings, these data can be used to improve system and energy performance, detect and diagnose faults, and facilitate maintenance. Building data are not only of interest to owners and operators of the building systems, however. Owners and operators of similar buildings, manufacturers of building systems, utilities, and city agencies also have interesting use cases. Energy data can, for example, be used to verify the performance of energy-conservation measures, issue renewableenergy certificates, and financially settle grid services. Increased data sharing, however, significantly expands the cyber-attack surface, creating new challenges. In this work, blockchain is explored as an option for ensuring the integrity of data that are shared by lighting and other building systems. Blockchain fundamentals and variants are briefly reviewed, and value propositions relevant to building systems are discussed. A recently developed blockchain applicability framework (BAF) that builds upon and addresses the limitations of previous applicability models is also briefly reviewed. The BAF is used to assess the suitability of blockchain over other technologies or approaches for building-data applications, using emerging connected lighting systems as an example use case.

blockchain, connected lighting system, data integr↗

Enabling Scalable and Extensible Memory-mapped Datastores in Userspace

Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Furthermore, through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9 × . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.

97 MATHEMATICS AND COMPUTING↗

Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA

The extreme-scale computing landscape is increasingly dominated by GPU-accelerated systems. At the same time, in-situ workflows that employ memory-to-memory inter-application data exchanges have emerged as an effective approach for leveraging these extreme-scale systems. In the case of GPUs, GPUDirect RDMA enables third-party devices, such as network interface cards, to access GPU memory directly and has been adopted for intra-application communications across GPUs. In this paper, we present an interoperable framework for GPU-based in-situ workflows that optimizes data movement using GPUDirect RDMA. Specifically, we analyze the characteristics of the possible data movement pathways between GPUs from an in-situ workflow perspective, and design a strategy that maximizes throughput. Furthermore, we implement this approach as an extension of the DataSpaces data staging service, and experimentally evaluate its performance and scalability on a current leadership GPU cluster. The performance results show that the proposed design reduces data-movement time by up to 53% and 40% for the sender and receiver, respectively, and maintains excellent scalability for up to 256 GPUs.

Zhang, Bo↗

CANDELS: THE COSMIC ASSEMBLY NEAR-INFRARED DEEP EXTRAGALACTIC LEGACY SURVEY-THE HUBBLE SPACE TELESCOPE OBSERVATIONS, IMAGING DATA PRODUCTS, AND MOSAICS

This paper describes the Hubble Space Telescope imaging data products and data reduction procedures for the Cosmic Assembly Near-infrared Deep Extragalactic Legacy Survey (CANDELS). This survey is designed to document the evolution of galaxies and black holes at z Almost-Equal-To 1.5-8, and to study Type Ia supernovae at z > 1.5. Five premier multi-wavelength sky regions are selected, each with extensive multi-wavelength observations. The primary CANDELS data consist of imaging obtained in the Wide Field Camera 3 infrared channel (WFC3/IR) and the WFC3 ultraviolet/optical channel, along with the Advanced Camera for Surveys (ACS). The CANDELS/Deep survey covers {approx}125 arcmin{sup 2} within GOODS-N and GOODS-S, while the remainder consists of the CANDELS/Wide survey, achieving a total of {approx}800 arcmin{sup 2} across GOODS and three additional fields (Extended Groth Strip, COSMOS, and Ultra-Deep Survey). We summarize the observational aspects of the survey as motivated by the scientific goals and present a detailed description of the data reduction procedures and products from the survey. Our data reduction methods utilize the most up-to-date calibration files and image combination procedures. We have paid special attention to correcting a range of instrumental effects, including charge transfer efficiency degradation for ACS, removal of electronic bias-striping present in ACS data after Servicing Mission 4, and persistence effects and other artifacts in WFC3/IR. For each field, we release mosaics for individual epochs and eventual mosaics containing data from all epochs combined, to facilitate photometric variability studies and the deepest possible photometry. A more detailed overview of the science goals and observational design of the survey are presented in a companion paper.

79 ASTRONOMY AND ASTROPHYSICS↗

5G Service Value Chain and Network Slicing Framework using Ecosystem Modeling, Agile Delivery, and User-Story Automation

The 5G technology allows telecommunication operators to provide services beyond data broadband for consumers, employing tailored use-cases for enterprise and vertical markets. As a result, there is an increasing need for a collaborative service development through a de-centralized value-chain, along with new business models and shorter time-to-market. In this paper, a novel, use-case driven framework for a modular 5G service design and delivery is proposed. The framework implements agile and as-a-service principles to provide 5G use-case customers access not only to service usage, but also to design and operation. The framework integrates with the 3rd Generation Partnership Project (3GPP) specifications by mapping network-slicing procedures to the designed ecosystem lifecycle operations. The framework allows for any-technology deployment, utilizing the available foundations for domain-specific automation. A structured collaboration between a 5G operator and a use-case user allows for the customization and improvement of the targeted use-case, where 5G network assets are updated with customer’s vertical components through programmable user-stories. Laboratory testing of a 5G network-slicing for a private-telephony and merchandise-tracking service is performed to validate the effectiveness of the proposed approach for real-world applications.

97 MATHEMATICS AND COMPUTING↗

The Modern Research Data Portal: A Design Pattern for Networked, Data-Intensive Science

Here we describe best practices for providing convenient, high-speed, secure access to large data via research data portals. We capture these best practices in a new design pattern, the Modern Research Data Portal, that disaggregates the traditional monolithic web-based data portal to achieve orders-of-magnitude increases in data transfer performance, support new deployment architectures that decouple control logic from data storage, and reduce development and operations costs. We introduce the design pattern; explain how it leverages high-performance Science DMZs and cloud-based data management services; review representative examples at research laboratories and universities, including both experimental facilities and supercomputer sites; describe how to leverage Python APIs for authentication, authorization, data transfer, and data sharing; and use coding examples to demonstrate how these APIs can be used to implement a range of research data portal capabilities. Sample code at a companion web site, https://docs.globus.org/mrdp, provides application skeletons that readers can adapt to realize their own research data portals.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

The Modern Research Data Portal: a design pattern for networked, data-intensive science

We describe best practices for providing convenient, high-speed, secure access to large data via research data portals. Here, we capture these best practices in a new design pattern, the Modern Research Data Portal, that disaggregates the traditional monolithic web-based data portal to achieve orders-of-magnitude increases in data transfer performance, support new deployment architectures that decouple control logic from data storage, and reduce development and operations costs. We introduce the design pattern; explain how it leverages high-performance data enclaves and cloud-based data management services; review representative examples at research laboratories and universities, including both experimental facilities and supercomputer sites; describe how to leverage Python APIs for authentication, authorization, data transfer, and data sharing; and use coding examples to demonstrate how these APIs can be used to implement a range of research data portal capabilities. Sample code at a companion web site, https://docs.globus.org/mrdp , provides application skeletons that readers can adapt to realize their own research data portals.

97 MATHEMATICS AND COMPUTING↗

Assessment of Methane Emissions – Impact of Using Natural Gas Engines in Unconventional Resource Development

Researchers at the Center for Alternative Fuels, Engines, and Emissions (CAFEE) completed a multi-year program under DE-FE0013689 entitled, “Assessing Fugitive Methane Emissions Impact Using Natural Gas Engines in Unconventional Resource Development.” When drilling activity was high and industry sought to lower operating costs and reduce emissions they began investing in dual fuel and dedicated natural gas engines to power unconventional well equipment. From a review of literature we determined that the prime-movers (or major fuel consumers) of unconventional well development were the service trucks (trucking), horizontal drilling rig (drilling) engines, and hydraulic stimulation pump (fracturing) engines. Based on early findings from on-road studies we assessed that conversion of prime movers to operate on natural gas could contribute to methane emissions associated with unconventional wells. As such, we collected significant in-use activity data from service trucks and in-use activity, fuel consumption, and gaseous emissions data from drilling and fracturing engines. Our findings confirmed that conversion of the prime movers to operate as dual fuel or dedicated natural gas – created an additional source of methane emissions. While some gaseous emissions were decreased from implementation of these technologies – methane and CO 2 equivalent emissions tended to increase, especially for non-road engines. The increases were highest for dual fuel engines due to methane slip from the exhaust and engine crankcase. Dedicated natural gas engines tended to have lower exhaust methane emissions but higher CO 2 emissions due to lower efficiency. Therefore, investing in currently available natural gas technologies for prime movers will increase the greenhouse gas footprint of the unconventional well development industry.

03 NATURAL GAS↗

Reducing Data Center Peak Cooling Demand and Energy Costs with Underground Thermal Energy Storage (UTES)

By recent estimates, data center energy demands are projected to consume between 6.7% and 12% of U.S. annual electricity generation by the year 2028, driven primarily by expanded demands from cloud services, big data analytics, and Artificial Intelligence (AI) (Shehabi et al., 2024). As much as 40% of data center total energy consumption are loads associated with the site infrastructure cooling systems, and these are often highly water consumptive (Aljbour et al., 2024). For energy system planners, this presents significant challenges to meeting and managing the anticipated loads, and especially the peak loads of projected data center deployments. Geothermal technologies offer two unique solutions to these challenges: 1) by serving loads through the deployment of new conventional and/or next-generation geothermal power technologies such as EGS and 2) through an often-overlooked opportunity to reduce data center peak cooling loads. The latter is the focus of this paper which explores Cold Underground Thermal Energy Storage ("Cold UTES") as an emerging industrial-scale geothermal cooling solution. This cooling solution is energy efficient, non-water-consumptive, and utilizes long duration energy storage (LDES) on both diurnal and seasonal time scales. Cold UTES has the potential to also function as a virtual power plant (VPP). The US Department of Energy's Geothermal Technologies Office is supporting R&D to understand the grid and system-wide value, costs, and impacts of deploying this emergent cooling solution at scale.

AI↗

DOE SciDAC's Earth System Grid Center for Enabling Technologies Final Report

The mission of the Earth System Grid Federation (ESGF) is to provide the worldwide climate-research community with access to the data, information, model codes, analysis tools, and intercomparison capabilities required to make sense of enormous climate data sets. Its specific goals are to (1) provide an easy-to-use and secure web-based data access environment for data sets; (2) add value to individual data sets by presenting them in the context of other data sets and tools for comparative analysis; (3) address the specific requirements of participating organizations with respect to bandwidth, access restrictions, and replication; (4) ensure that the data are readily accessible through the analysis and visualization tools used by the climate research community; and (5) transfer infrastructure advances to other domain areas. For the ESGF, the U.S. Department of Energy's (DOE's) Earth System Grid Center for Enabling Technologies (ESG-CET) team has led international development and delivered a production environment for managing and accessing ultra-scale climate data. This production environment includes multiple national and international climate projects (such as the Community Earth System Model and the Coupled Model Intercomparison Project), ocean model data (such as the Parallel Ocean Program), observation data (Atmospheric Radiation Measurement Best Estimate, Carbon Dioxide Information and Analysis Center, Atmospheric Infrared Sounder, etc.), and analysis and visualization tools, all serving a diverse user community. These data holdings and services are distributed across multiple ESG-CET sites (such as ANL, LANL, LBNL/NERSC, LLNL/PCMDI, NCAR, and ORNL) and at unfunded partner sites, such as the Australian National University National Computational Infrastructure, the British Atmospheric Data Centre, the National Oceanic and Atmospheric Administration Geophysical Fluid Dynamics Laboratory, the Max Planck Institute for Meteorology, the German Climate Computing Centre, the National Aeronautics and Space Administration Jet Propulsion Laboratory, and the National Oceanic and Atmospheric Administration. The ESGF software is distinguished from other collaborative knowledge systems in the climate community by its widespread adoption, federation capabilities, and broad developer base. It is the leading source for present climate data holdings, including the most important and largest data sets in the global-climate community, and - assuming its development continues - we expect it to be the leading source for future climate data holdings as well. Recently, ESG-CET extended its services beyond data-file access and delivery to include more detailed information products (scientific graphics, animations, etc.), secure binary data-access services (based upon the OPeNDAP protocol), and server-side analysis. The latter capabilities allow users to request data subsets transformed through commonly used analysis and intercomparison procedures. As we transition from development activities to production and operations, the ESG-CET team is tasked with making data available to all users seeking to understand, process, extract value from, visualize, and/or communicate it to others. This ongoing effort, though daunting in scope and complexity, will greatly magnify the value of numerical climate model outputs and climate observations for future national and international climate-assessment reports. The ESG-CET team also faces substantial technical challenges due to the rapidly increasing scale of climate simulation and observational data, which will grow, for example, from less than 50 terabytes for the last Intergovernmental Panel on Climate Change (IPCC) assessment to multiple Petabytes for the next IPCC assessment. In a world of exponential technological change and rapidly growing sophistication in climate data analysis, an infrastructure such as ESGF must constantly evolve if it is to remain relevant and useful. Regretfully, we submit our final report at the end of project funding. To continue to serve the climate-science community, we are currently seeking additional funding. Such funding would allow us to maintain and enhance ESGF production and operation of this vital endeavor of cataloging, serving, and analyzing ultra-scale climate science data. At this time, the entire ESG-CET team would like to take this opportunity to sincerely thank our funding agencies in the DOE Scientific Discovery through Advanced Computing (SciDAC) program and the Office of Biological and Environmental Research (OBER) - as well as our national and international collaborators, stakeholders, and partners - for allowing us to work with you and serve the community these past several years.

54 ENVIRONMENTAL SCIENCES↗

Translator Plan: A Coordinated Vision for Fiscal Years 2023-2025

Translators serve a unique role in the U.S. Department of Energy (DOE)’s Atmospheric Radiation Measurement (ARM) user facility, offering scientific input through various leadership and service roles. The Translators direct the creation of value-added products (VAPs) and analysis tools that make ARM measurements more accessible to the scientific community. Translators also serve as liaisons between users and the ARM infrastructure, collecting information about priorities and communicating ARM data and services. A key group focus is supporting the DOE Atmospheric System Research (ASR) program scientists and ASR’s efforts towards a process-level understanding of cloud-aerosol interactions, and in reducing uncertainty in global climate model projections. The ARM Translator Group (Table 1) consists of the five Translators, a representative of software development, and one from the Data Quality Office (DQO). Additionally, the ARM Engineering and Process Manager participates in this group and provides input and direction from ARM and its programmatic priorities.

54 ENVIRONMENTAL SCIENCES↗

The Exascale Framework for High Fidelity coupled Simulations (EFFIS): Enabling whole device modeling in fusion science

We present the Exascale Framework for High Fidelity coupled Simulations (EFFIS), a workflow and code coupling framework developed as part of the Whole Device Modeling Application (WDMApp) in the Exascale Computing Project. EFFIS consists of a library, command line utilities, and a collection of run-time daemons. Together, these software products enable users to easily compose and execute workflows that include: strong or weak coupling, in situ (or offline) analysis/visualization/monitoring, command-and-control actions, remote dashboard integration, and more. We describe WDMApp physics coupling cases and computer science requirements that motivate the design of the EFFIS framework. Furthermore, we explain the essential enabling technology that EFFIS leverages: ADIOS for performant data movement, PerfStubs/TAU for performance monitoring, and an advanced COUPLER for transforming coupling data from its native format to the representation needed by another application. Finally, we demonstrate EFFIS using coupled multi-simulation WDMApp workflows and exemplify how the framework supports the project’s needs. We show that EFFIS and its associated services for data movement, visualization, and performance collection does not introduce appreciable overhead to the WDMApp workflow and that the resource-dominant application’s idle time while waiting for data is minimal.

97 MATHEMATICS AND COMPUTING↗

NEMESIS: Exoplanet Transit Survey of Nearby M-dwarfs in TESS FFIs. I

In this work, we present an analysis of 33,054 M-dwarf stars, located within 100 parsecs, via the Transiting Exoplanet Survey Satellite (TESS) full-frame images (FFIs) of observed sectors 1–5. We present a new pipeline called NEMESIS, developed to extract detrended photometry, and to perform transit searches of single-sector data in TESS FFIs. As many M-dwarfs are faint, and are not observed with a two-minute cadence by TESS, FFI transit surveys can provide an empirical validation of how many planets are missed, using the 30-minute cadence data. In this work, we detect 183 threshold crossing events, and present 29 candidate planets for sectors 1–5, 24 of which are new detections. Our sample contains orbital periods ranging from 1.25 to 6.84 days, and planetary radii from 1.26 to 5.31 R {sub ⊕}. With the addition of our new planet candidate detections, along with detections previously observed in sectors 1–5, we calculate an integrated occurrence rate of 2.49 ± 1.58 planets per star, for the period range ∈ [1, 9] days, and planet radius range ∈ [0.5,11] R {sub ⊕}. We project an estimated yield of 122 ± 11 transit detections of nearby M-dwarfs. Of our new candidates, 23 have signal-to-noise ratios >7, transmission spectroscopy metrics >38, and emission spectroscopy metrics >10. We present all of our data products for our planet candidates via the Filtergraph data visualization service, located at https://filtergraph.com/NEMESIS.

47 OTHER INSTRUMENTATION↗

Redesigning the DOE Data Explorer to embed dataset relationships at the point of search and to reflect landing page organization

Scientific research is producing ever-increasing amounts of data. Organizing and reflecting relationships across data collections, datasets, publications, and other research objects are essential functionalities of the modern science environment, yet challenging to implement. Landing pages are often used for providing ‘big picture’ contextual frameworks for datasets and data collections, and many large-volume data holders are utilizing them in thoughtful, creative ways. The benefits of their organizational efforts, however, are not realized unless the user eventually sees the landing page at the end point of their search. What if that organization and ‘big picture’ context could benefit the user at the beginning of the search? That is a challenging approach, but The Department of Energy’s (DOE) Office of Scientific and Technical Information (OSTI) is redesigning the database functionality of the DOE Data Explorer (DDE) with that goal in mind. Phase I is focused on redesigning the DDE database to leverage relationships between two existing distinct populations in DDE, data Projects and individual Datasets, and then adding a third intermediate population, data Collections. Mapped, structured linkages, designed to show user relationships, will allow users to make informed search choices. These linkages will be sustainable and scalable, created automatically with the use of new metadata fields and existing authorities. Phase II will study selected DOE Data ID Service clients, analyzing how their landing pages are organized, and how that organization might be used to improve DDE search capabilities. At the heart of both phases is the realization that adding more metadata information for cross-referencing may require additional effort for data scientists. Finally, OSTI’s approach seeks to leverage existing metadata and landing page intelligence without imposing an additional burden on the data creators.

97 MATHEMATICS AND COMPUTING↗

Finding Tropical Cyclones on a Cloud Computing Cluster: Using Parallel Virtualization for Large-Scale Climate Simulation Analysis

Extensive computing power has been used to tackle issues such as climate changes, fusion energy, and other pressing scientific challenges. These computations produce a tremendous amount of data; however, many of the data analysis programs currently only run a single processor. In this work, we explore the possibility of using the emerging cloud computing platform to parallelize such sequential data analysis tasks. As a proof of concept, we wrap a program for analyzing trends of tropical cyclones in a set of virtual machines (VMs). This approach allows the user to keep their familiar data analysis environment in the VMs, while we provide the coordination and data transfer services to ensure the necessary input and output are directed to the desired locations. This work extensively exercises the networking capability of the cloud computing systems and has revealed a number of weaknesses in the current cloud system software. In our tests, we are able to scale the parallel data analysis job to a modest number of VMs and achieve a speedup that is comparable to running the same analysis task using MPI. However, compared to MPI based parallelization, the cloud-based approach has a number of advantages. The cloud-based approach is more flexible because the VMs can capture arbitrary software dependencies without requiring the user to rewrite their programs. The cloud-based approach is also more resilient to failure; as long as a single VM is running, it can make progress while as soon as one MPI node fails the whole analysis job fails. In short, this initial work demonstrates that a cloud computing system is a viable platform for distributed scientific data analyses traditionally conducted on dedicated supercomputing systems.

97 MATHEMATICS AND COMPUTING↗