Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “File transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2002

MDP: Reliable File Transfer for Space Missions

This paper presents work being done at NASA/GSFC (Goddard Space Flight Center) by the Operating Missions as Nodes on the Internet (OMNI) project to demonstrate the application of the Multicast Dissemination Protocol (MDP) to space missions to reliably transfer files. This work builds on previous work by the OMNI project to apply Internet communication technologies to space communication. The goal of this effort is to provide an inexpensive, reliable, standard, and interoperable mechanism for transferring files in the space communication environment. Limited bandwidth, noise, delay, intermittent connectivity, link asymmetry, and one-way links are all possible issues for space missions. Although these are link-layer issues, they can have a profound effect on the performance of transport and application level protocols. MDP, a UDP (User Datagram Protocol)-based reliable file transfer protocol, was designed for multicast environments which have to address these same issues, and it has done so successfully. Developed by the Naval Research Lab in the mid 1990s, MDP is now in daily use by both the US Post Office and the DoD (Department of Defense). This paper describes the use of MDP to provide automated end-to-end data flow for space missions. It examines the results of a parametric study of MDP in a simulated space link environment and discusses the results in terms of their implications for space missions. Lessons learned are addressed, which suggest minor enhancements to the MDP user interface to add specific features for space mission requirements, such as dynamic control of data rate, and a checkpoint/resume capability. These are features that are provided for in the protocol, but are not implemented in the sample MDP application that was provided. A brief look is also taken at the status of standardization. A version of MDP known as NORM (Nack Oriented Reliable Multicast) is in the process of becoming an IETF (Internet Engineering Task Force) standard.

Rash, James↗
NASA NTRS2002

MDP: Reliable File Transfer for Space Missions

This paper presents work being done at NASA/GSFC by the Operating Missions as Nodes on the Internet (OMNI) project to demonstrate the application of the Multicast Dissemination Protocol (MDP) to space missions to reliably transfer files. This work builds on previous work by the OMNI project to apply Internet communication technologies to space communication. The goal of this effort is to provide an inexpensive, reliable, standard, and interoperable mechanism for transferring files in the space communication environment. Limited bandwidth, noise, delay, intermittent connectivity, link asymmetry, and one-way links are all possible issues for space missions. Although these are link-layer issues, they can have a profound effect on the performance of transport and application level protocols. MDP, a UDP-based reliable file transfer protocol, was designed for multicast environments which have to address these same issues, and it has done so successfully. Developed by the Naval Research Lab in the mid 1990's, MDP is now in daily use by both the US Post Office and the DoD. This paper describes the use of MDP to provide automated end-to-end data flow for space missions. It examines the results of a parametric study of MDP in a simulated space link environment and discusses the results in terms of their implications for space missions. Lessons learned are addressed, which suggest minor enhancements to the MDP user interface to add specific features for space mission requirements, such as dynamic control of data rate, and a checkpoint/resume capability. These are features that are provided for in the protocol, but are not implemented in the sample MDP application that was provided. A brief look is also taken at the status of standardization. A version of MDP known as NORM (Neck Oriented Reliable Multicast) is in the process of becoming an IETF standard.

Rash, James↗
DOE OSTI2019

File transfers between machines without target CPU intervention

Techniques for transferring files between machines include creating a zero-length target file on non-volatile storage, truncating the file to a desired size, and allocating storage on the non-volatile storage for each block of the target file. The technique also includes determining a logical block address (LBA) for each location in the target file. The technique further includes sending a request to an input/output (I/O) node to transfer a source file to the non-volatile storage, where the request includes a mapping between the LBAs and file offsets. The technique includes opening the source file and a block device at the I/O node. The technique further includes reading each block from the source file and writing each block to the target file on the non-volatile storage utilizing the block device, and then closing the source file and the block device.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Building a Wide-Area File Transfer Performance Predictor: An Empirical Study

Wide-area data transfer is central to geographically distributed scientific workflows. Faster delivery of data is important for these workflows. Predictability is equally (or even more) important. With the goal of providing a reasonably accurate estimate of data transfer time to improve resource allocation & scheduling for workflows and enable end-to-end data transfer optimization, we apply machine learning methods to develop predictive models for data transfer times over a variety of wide area networks. To build and evaluate these models, we use 201,388 transfers, involving 759 million files totaling 9 PB transferred, over 115 heavily used source-destination pairs (“edges”) between 135 unique endpoints. We evaluate the models for different retraining frequencies and different window size of history data. In the best case, the resulting models have a median prediction error of ≤21% for 50% of the edges, and ≤32% for 75% of the edges. We present a detailed analysis of these results that provides insights into the cause of some of the high errors. We envision that the performance predictor will be informative for scheduling geo-distributed workflows. The insights also suggest obvious directions for both further analysis and transfer service optimization.

Liu, Zhengchun↗
DOE OSTI2019

Unification of Monitoring in the Fermi File Transfer Service

Written in Python, the Fermi File Transfer Service (FTS) is a sitewide program used by Fermilab to safely transfer files to storage. With the scale of FTS, monitoring its status has become increasingly important. By using Kafka-Python, FNAL Landscape, and Elasticsearch, the project's goal is to provide enhanced remote monitoring to compliment FTS's pre-existing monitoring. This will enable new ways to visually graph and interpret logs in a non-local environment. The associated poster briefly describes the changes made, details about Elasticsearch, and illustrates the monitoring data received thus far.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2005

Fail-over file transfer process

The present invention provides a fail-over file transfer process to handle data file transfer when the transfer is unsuccessful in order to avoid unnecessary network congestion and enhance reliability in an automated data file transfer system. If a file cannot be delivered after attempting to send the file to a receiver up to a preset number of times, and the receiver has indicated the availability of other backup receiving locations, then the file delivery is automatically attempted to one of the backup receiving locations up to the preset number of times. Failure of the file transfer to one of the backup receiving locations results in a failure notification being sent to the receiver, and the receiver may retrieve the file from the location indicated in the failure notification when ready.

Semancik, Susan K.↗
DOE OSTI2019

Measurements and analytics of wide-area file transfers over dedicated connections

Distributed scientific and big-data computations are becoming increasingly dependent on access to remote files. Wide-area file transfers are supported by two basic schemes: (i) application-level tools, such as GridFTP, that provide transport services between file systems housed at geographically separated sites, and (ii) file systems mounted over wide-area networks, using mechanisms such as LNet routers that make them transparently available. In both cases, the file transfer performance critically depends on the configuration consisting of host, file, IO, and disk subsystems, which are complex by themselves, as well as on their complex compositions implemented using buffers and IO-network data transitions. We present extensive file transfer rate measurements collected over dedicated 10 Gbps connections with 0-366 ms round-trip times, using GridFTP and XDD file transfer tools, and Lustre file system extended over wide-area networks using LNet routers. Our test configurations are composed of: three types of host systems; XFS, Lustre, and ext3 file systems; and Ethernet and SONET wide-area connections. We present analytics based on the convexity-concavity of throughput profiles which provide insights into throughput and its superior or inferior trend compared to linear interpolations. We propose the utilization-concavity coefficient, a scalar metric that characterizes the overall performance of any file transfer method consisting of specific configuration and scheme. Our results enable performance optimizations by highlighting the significant roles of (i) buffer sizes and parallelism in GridFTP and XDD, and (ii) buffer utilization and credit mechanism in LNet routers.

Rao, Nageswara↗
NASA NTRS1996

Data Transfer Efficiency Over Satellite Circuits Using a Multi-Socket Extension to the File Transfer Protocol (FTP)

In several experiments using NASA's Advanced Communications Technology Satellite (ACTS), investigators have reported disappointing throughput using the transmission control protocol/Internet protocol (TCP/IP) protocol suite over 1.536Mbit/sec (T1) satellite circuits. A detailed analysis of file transfer protocol (FTP) file transfers reveals that both the TCP window size and the TCP 'slow starter' algorithm contribute to the observed limits in throughput. In this paper we summarize the experimental and and theoretical analysis of the throughput limit imposed by TCP on the satellite circuit. We then discuss in detail the implementation of a multi-socket FTP, XFTP client and server. XFTP has been tested using the ACTS system. Finally, we discuss a preliminary set of tests on a link with non-zero bit error rates. XFTP shows promising performance under these conditions, suggesting the possibility that a multi-socket application may be less effected by bit errors than a single, large-window TCP connection.

Allman, Mark↗
NASA NTRS1984

Asynchronous file transfer to IBM PC's

The Asynchronous File Transfer System is used for interactively selecting and tramsmitting text files from the Langley Research Center's Business Data Systems Division (BDSD) host processor to an IBM Personal Computer. The IBM asynchronous communications support package is used for handling the communications on the personal computer. An application program (NATURAL, COBOL, etc.) is used for selecting and formatting the records to be transmitted, and three subroutine modules residing on the BDSD host processor are used for interfacing the application program and the communications software. Each record transmitted to the Personal Computer must be in the standard ASCII format. This record format is directly accessible by BASIC and many of the personal computer software packages provide utility programs for converting them to the format required for the particular package in question.

Hoerger, J.↗
NASA NTRS1987

Performance analysis of an optimal file transfer protocol for integrated mobile satellite services

A file-transfer protocol based on the selective-repeat automatic-retransmission-request (ARQ) scheme has been developed for the Mobile Satellite Experiment (MSAT-X). In this protocol, each file is divided into packets, which are error-detected and acknowledge individually. A file can be transferred in either the open- or closed-end connection. In the open-end connection, the file is transmitted continuously according to the selective-repeat scheme until all packets are positively acknowledged; in the closed-end connection, each retransmission will initiate its own connection request. The relationship between the packet size and the average end-to-end delay in the mobile fading environment is investigated. The communications channel uses trellis-coded modulation with symbol interleaving, packet replication, and error detection. The packet error performance in such a channel is evaluated by simulation, and the optimal packet size for various operating conditions is derived, using the simulated results. It is shown that for a symbol-energy-to-noise ratio of 12 dB, each file should be divided into packets of length between 256 bits and 512 bits (excluding the overhead) to minimize the average end-to-end delay.

Wang, Charles C.↗
NASA NTRS1986

Configurations for file transfer protocol error protection

This note considers codes for computer file transfer protocols when only upper-case ASCII characters are used as control characters. Such a restriction may be necessary to avoid dependence on unique machine features and to promote portability. If ten control functions are needed, a number used in a typical sufficiently rich protocol, we seek a subset of ten upper-case ASCII characters with good distance properties. The control functions form themselves naturally into three groups. Recognizing that only even distances can occur, the need to make the intragroup minimum distances as large as possible, and the number of control characters have distance 2 as small as possible is emphasized. The optimum solution to the above problem is found as well as an assignment that attains the optimum. The codes are essentially unique.

Posner, E. C.↗
NASA NTRS1988

An operational open-end file transfer protocol for mobile satellite communications

This paper describes an operational open-end file transfer protocol which includes the connecting procedure, data transfer, and relinquishment procedure for mobile satellite communications. The protocol makes use of the frame level and packet level formats of the X.25 standard for the data link layer and network layer, respectively. The structure of a testbed for experimental simulation of this protocol over a mobile fading channel is also introduced.

Wang, Charles↗
NASA NTRS2004

Software Implements a Space-Mission File-Transfer Protocol

CFDP is a computer program that implements the CCSDS (Consultative Committee for Space Data Systems) File Delivery Protocol, which is an international standard for automatic, reliable transfers of files of data between locations on Earth and in outer space. CFDP administers concurrent file transfers in both directions, delivery of data out of transmission order, reliable and unreliable transmission modes, and automatic retransmission of lost or corrupted data by use of one or more of several lost-segment-detection modes. The program also implements several data-integrity measures, including file checksums and optional cyclic redundancy checks for each protocol data unit. The metadata accompanying each file can include messages to users application programs and commands for operating on remote file systems.

Rundstrom, Kathleen↗
NASA NTRS2004

Automated File Transfer and Storage Management Concepts for Space

This presentation will summarize work that has been done to prototype and analyze approaches for automated file transfer and storage management for space missions. The concepts were prototyped in an environment with data files being generated at the target mission rates and stored in onboard files. The space-to-ground link was implemented using a channel simulator to introduce representative mission delays and errors. The system was operated for days with data files building up on the spacecraft and periodically being transferred to ground storage during a limited contact time. Overall performance was measured to identify limits under which the entire data volume could be transferred automatically while still fitting into the mission s limited contact time. The overall concepts, measurements, and results will be presented.

Hogie, Keith↗
NASA NTRS2010

Large File Transfers from Space Using Multiple Ground Terminals and Delay-Tolerant Networking

We use Delay-Tolerant Networking (DTN) to break control loops between space-ground communication links and ground-ground communication links to increase overall file delivery efficiency, as well as to enable large files to be proactively fragmented and received across multiple ground stations. DTN proactive fragmentation and reactive fragmentation were demonstrated from the UK-DMC satellite using two independent ground stations. The files were reassembled at a bundle agent, located at Glenn Research Center in Cleveland Ohio. The first space-based demonstration of this occurred on September 30 and October 1, 2009. This paper details those experiments. Communication, delay-tolerant networking, DTN, satellite, Internet, protocols, bundle, IP, TCP.

Ivancic, William D.↗
NASA NTRS2006

Transferring Files Between the Deep Impact Spacecrafts and the Ground Data System Using the CCSDS File Delivery Protocol (CFDP): A Case Study

The CCSDS File Delivery Protocol (CFDP) Standard could reshape ground support architectures by enabling applications to communicate over the space link using reliable-symmetric transport services. JPL utilized the CFDP standard to support the Deep Impact Mission. The architecture was based on layering the CFDP applications on top of the CCSDS Space Link Extension Services for data transport from the mission control centers to the ground stations. On July 4, 2005 at 1:52 A.M. EDT, the Deep Impact impactor successfully collided with comet Tempel 1. During the final 48 hours prior to impact, over 300 files were uplinked to the spacecraft, while over 6 thousand files were downlinked from the spacecraft using the CFDP. This paper uses the Deep Impact Mission as a case study in a discussion of the CFDP architecture, Deep Impact Mission requirements, and design for integrating the CFDP into the JPL deep space support services. Issues and recommendations for future missions using CFDP are also provided.

CCSDS File Delivery Protocol (CFDP)↗