Engineering Papers⌕ Search

Engineering topics

Foster, I.

Publications and source records attributed to Foster, I..

On Analytics of File Transfer Rates over Dedicated Wide-Area Connections

File transfers between the decentralized storage sites over dedicated wide-area connections are becoming increasingly important in high-performance computing and big data scenarios. Designing such scientific workflows for large file transfers is extremely challenging as they depend on the file, I/O, host, and local- and wide-area network subsystems, and their interactions. To gain insights into file-transfer rate profiles, we develop polynomial, bagging, and boosting regression models for Lustre and XFS file transfer measurements, which are collected using XDD over a suite of 10 Gbps connections with 0-366 ms round trip times (RTTs). In addition to overall trends and analytics, these regressions also provide file-transfer rate estimates for RTTs and number of parallel flows at which measurements might not have been collected. They show that bagging and boosting techniques provide closer data fits than the polynomial regression. We develop probabilistic bounds on the generalization error of these methods, which combined with the cross-validation error establish that former two are more accurate estimators than the polynomial regression. In addition, we present a method to efficiently determine the number of parallel flows to achieve a peak file-transfer rate using fewer than full sweep measurements; in our measurements, the peak is achieved in 96% of cases with 15-25% of measurements of a full sweep.

Sen, Satyabrata↗

Adaptive QoS for Data Transfers using Software-Defined Networking

Software-defined networking (SDN) has demonstrated tremendous potential for altering the networking landscape in both academia andindustry. While traffic engineering as network intelligence in classic technologies have received considerable attention from the researchcommunity, traffic engineering in the SDN space, particularly in regards to quality of service (QoS), remains relatively unexplored.While recent efforts have begun to show how SDN might be leverage to meet the complex traffic-shaping needs of modern networks, there is still much that can be done to illustrate its potential, and in particular how it could prevent degradation of service for all QoStraffic. We propose methods to allow other flows to use bandwidth unused by high-priority transfers, while ensuring that bandwidthassigned to those high-priority transfers is available when needed. We present fine-tuned control over bandwidth allocation that achieves an efficient combination of both the differentiated service (DiffServ)-style classification and the integrated service (IntServ)-style perflowconsiderations. We also show how this method may be used over a wide area network (WAN) to optimally adjust bandwidth allocations when high-priority data transfers are not using their allocated bandwidth.

Boley, Josh↗

Advance Reservation Access Control Using Software-Defined Networking and Tokens

Advance reservation systems allow users to reserve dedicated bandwidth connection resources from an advanced high-speed network. A common use case for such systems is data transfers in distributed science environments in which a user wants exclusive access to the reservation. However, current advance network reservation methods cannot ensure exclusive access of a network reservation to the specific flow for which the user made the reservation. We present here a novel network architecture that addresses this limitation and ensures that a reservation is used only by the intended flow. We achieve this by leveraging software-defined networking (SDN) and token-based authorization. We use SDN to orchestrate and automate the reservation of networking resources, end-to-end and across multiple administrativedomains, and tokens to create a strong binding between the user or application that requested the reservation and the flows provisioned by SDN.

Kettimuthu, R.↗

High-Performance data flows using analytical models and measurements

The combination of analytical models and measurements provide practical configurations and parameters to achieve high data transport rates: (a) buffer sizes and number of parallel streams for improved memory and file transfer rates, (b) Hamilton and Scalable TCP congestion control modules for memory transfers in place of default CUBIC, and (c) direct IO mode for Lustre file systems for wide-area transfers. Conventional parameter selection using full sweeps is impractical in many cases since it takes months. By exploiting the unimodality of throughput profiles, we developed the d-w method that significantly reduces the number of measurements needed for parameter identification. This heuristic method was effective in practice in reducing the measurements by about 90% for Lustre and XFS file transfers.

Rao, Nageswara S↗

The global gridded crop model intercomparison: Data and modeling protocols for Phase 1 (v1.0)

We present protocols and input data for Phase 1 of the Global Gridded Crop Model Intercomparison, a project of the Agricultural Model Intercomparison and Improvement Project (AgMIP). The project consist of global simulations of yields, phenologies, and many land-surface fluxes using 12–15 modeling groups for many crops, climate forcing data sets, and scenarios over the historical period from 1948 to 2012. The primary outcomes of the project include (1) a detailed comparison of the major differences and similarities among global models commonly used for large-scale climate impact assessment, (2) an evaluation of model and ensemble hindcasting skill, (3) quantification of key uncertainties from climate input data, model choice, and other sources, and (4) a multi-model analysis of the agricultural impacts of large-scale climate extremes from the historical record.

54 ENVIRONMENTAL SCIENCES↗

Climate Science for a Sustainable Energy Future Test Bed and Data Infrastructure Final Report

The collaborative Climate Science for a Sustainable Energy Future (CSSEF) project started in July 2011 with the goal of accelerating the development of climate model components (i.e., atmosphere, ocean and sea ice, and land surface) and enhancing their predictive capabilities while incorporating uncertainty quantification (UQ). This effort required accessing and converting observational data sets into specialized model testing and verification data sets and building a model development test bed, where model components and sub-models can be rapidly evaluated. CSSEF’s prototype test bed demonstrated, how an integrated testbed could eliminate tedious activities associated with model development and evaluation, by providing the capability to constantly compare model output—where scientists store, acquire, reformat, regrid, and analyze data sets one-by-one—to observational measurements in a controlled test bed.

54 ENVIRONMENTAL SCIENCES↗