SEARCH · Engineering Papers

Results for “File transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

All sources NASA NTRS DOE OSTI

NASA NTRS2000

Internet-based transfer of cardiac ultrasound images

A drawback to large-scale multicentre studies is the time required for the centralized evaluation of diagnostic images. We evaluated the feasibility of digital transfer of echocardiographic images to a central laboratory for rapid and accurate interpretation. Ten patients undergoing trans-oesophageal echocardiographic scanning at three sites had representative single images and multiframe loops stored digitally. The images were analysed in the ordinary way. All images were then transferred via the Internet to a central laboratory and reanalysed by a different observer. The file sizes were 1.5-72 MByte and the transfer rates achieved were 0.6-4.8 Mbit/min. Quantitative measurements were similar between most on-site and central laboratory measurements (all P > 0.25), although measurements differed for left atrial width and pulmonary venous systolic velocities (both P < 0.05). Digital transfer of echocardiographic images and data to a central laboratory may be useful for multicentre trials.

NASA Discipline Cardiopulmonary↗

NASA NTRS2012

An Evaluation of Protocol Enhancing Proxies and File Transport Protocols for Satellite Communication

NASA is utilizing Global Hawk aircraft in high-altitude, long-duration Earth science missions. Communications with the onboard research equipment and sensors (the science payload) is via Ku-Band radio utilizing satellites in geostationary orbits. All payload communications use standard Internet Protocols and routing, and much of the data to be transferred is comprised of very large files. The science community is interested in fully utilizing these communication links to retrieve data as quickly and reliably as possible. A test bed was developed at NASA Ames to evaluate modern transport protocols as well as Protocol Enhancing Proxies (PEPs) to determine what tools best fit the needs of the science community. This paper describes the test bed used, the protocols, the PEPs that were evaluated, the particular tests performed and the results and conclusions.

Finch, Patrick Eugene↗

DOE OSTI2019

High-performance End-to-End Integrity Verification on Big Data Transfer

The scale of scientific data generated by experimental facilities and simulations in high-performance computing facilities has been proliferating with the emergence of IoT-based big data. In many cases, this data must be transmitted rapidly and reliably to remote facilities for storage, analysis, or sharing, for the Internet of Things (IoT) applications. Simultaneously, IoT data can be verified using a checksum after the data has been written to the disk at the destination to ensure its integrity. However, this end-to-end integrity verification inevitably creates overheads (extra disk I/O and more computation). Thus, the overall data transfer time increases. In this article, we evaluate strategies to maximize the overlap between data transfer and checksum computation for astronomical observation data. Specifically, we examine file-level and block-level (with various block sizes) pipelining to overlap data transfer and checksum computation. We analyze these pipelining approaches in the context of GridFTP, a widely used protocol for scientific data transfers. Theoretical analysis and experiments are conducted to evaluate our methods. The results show that block-level pipelining is effective in maximizing the overlap mentioned above, and can improve the overall data transfer time with end-to-end integrity verification by up to 70% compared to the sequential execution of transfer and checksum, and by up to 60% compared to file-level pipelining.

data integrity↗

DOE OSTI2020

GPU Direct I/O with HDF5

Exascale HPC systems are being designed with accelerators, such as GPUs, to accelerate parts of applications. In machine learning workloads as well as large-scale simulations that use GPUs as accelerators, the CPU (or host) memory is currently used as a buffer for data transfers between GPU (or device) memory and the file system. If the CPU does not need to operate on the data, then this is sub-optimal because it wastes host memory by reserving space for duplicated data. Furthermore, this “bounce buffer” approach wastes CPU cycles spent on transferring data. A new technique, NVIDIA GPUDirect Storage (GDS), can eliminate the need to use the host memory as a bounce buffer. Thereby, it becomes possible to transfer data directly between the device memory and the file system. This direct data path shortens latency by omitting the extra copy and enables higher-bandwidth. To take full advantage of GDS in existing applications, it is necessary to provide support with existing I/O libraries, such as HDF5 and MPI-IO, which are heavily used in applications. In this paper, we describe our effort of integrating GDS with HDF5, the top I/O library at NERSC and at DOE leadership computing facilities. We design and implement this integration using a HDF5 Virtual File Driver (VFD). The GDS VFD provides a file system abstraction to the application that allows HDF5 applications to perform I/O without the need to move data between CPUs and GPUs explicitly. We compare performance of the HDF5 GDS VFD with explicit data movement approaches and demonstrate superior performance with the GDS method.

Ravi, J↗

NASA NTRS1991

Surface-Shading Program

Surface Shading computer program, SURF, developed in support of work of Computational Fluid Dynamics (CFD). Part of CFD graphics software. Accepts input in form of grid and solution files otherwise normally fed to Ames Research Center's version of PLOT3D software from "flow solver" programs. Interactively creates wire-frame, shaded, and function-map parts viewed and then transferred to Ames Research Graphic (ARCGRAPH) standard files, animated with GAS (COSMIC Program ARC-12379). Written in C programming language and requires Silicon Graphics "include" files (e.g., stdio.h, gl.h).

Plessel, Todd↗

DOE OSTI2024

NSTXU Diagnostic Disruption Dynamic Loading Represented by Response Spectra

This article presents the results of transient dynamic simulations of loads due to disruption eddy currents on the NSTXU vacuum vessel. Dynamic loading at diagnostic mounting locations is expressed as response spectra derived from the time history results of the dynamic structural simulations of a variety of disruption scenarios. The disruption simulations draw on a history of the project assessments of worst case disruptions for specific components. Major efforts to assess disruption loading have included the vacuum vessel which is the major structural support for the machine, as well as the passive plates (PPs), high harmonic fast wave (HHFW) antenna, and centerstack casing. Each one of these efforts included transient electromagnetic simulations producing time-dependent eddy current Lorentz loads (and in some cases halo loads) which then were applied to time-dependent structural dynamic analyses intended to obtain the proper dynamic amplification factors. In some instances, the EM model and structural model were identical allowing direct transfer of EM forces to the structural model. In other cases, the EM and structural model were not identical and the vector potential (VP) transfer method was used. The results files from these analyses were available (or re-run) to post process in ANSYS Classic time history postprocessor. In conclusion, the ANSYS command is used to create response spectra from time history data at desired points on the vessel.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

NASA NTRS2015

Deploying the NASA Meter Class Autonomous Telescope (MCAT) on Ascension Island

NASA has successfully constructed the 1.3m Meter Class Autonomous Telescope (MCAT) facility on Ascension Island in the South Atlantic Ocean. MCAT is an optical telescope designed specifically to collect ground-based data for the statistical characterization of orbital debris ranging from Low Earth Orbit (LEO) through Middle Earth Orbits (MEO) and beyond to Geo Transfer and Geosynchronous Orbits (GTO/GEO). The location of Ascension Island has two distinct advantages. First, the near-equatorial location fills a significant longitudinal gap in the Ground-based Electro-Optical Deep Space Surveillance (GEODSS) network of telescopes, and second, it allows access to objects in Low Inclination Low-Earth Orbits (LILO). The MCAT facility will be controlled by a sophisticated software suite that operates the dome and telescope, assesses sky and weather conditions, conducts all necessary calibrations, defines an observing strategy (as dictated by weather, sky conditions and the observing plan for the night), and carries out the observations. It then reduces the collected data via four primary observing modes ranging from tracking previously cataloged objects to conducting general surveys for detecting uncorrelated debris. Nightly observing plans, as well as the resulting text file of reduced data, will be transferred to and from Ascension, respectively, via a satellite connection. Post-processing occurs at NASA Johnson Space Center. Construction began in September, 2014 with dome and telescope installation occurring in April through early June, 2015. First light was achieved in June, 2015. Acceptance testing, full commissioning, and calibration of this soon-to-be fully autonomous system commenced in summer 2015. The initial characterization of the system from these tests is presented herein.

Lederer, S. M.↗

NASA NTRS1991

Data storage and retrieval system

The Data Storage and Retrieval System (DSRS) consists of off-the-shelf system components integrated as a file server supporting very large files. These files are on the order of one gigabyte of data per file, although smaller files on the order of one megabyte can be accommodated as well. For instance, one gigabyte of data occupies approximately six 9 track tape reels (recorded at 6250 bpi). Due to this large volume of media, it was desirable to shrink the size of the proposed media to a single portable cassette. In addition to large size, a key requirement was that the data needs to be transferred to a (VME based) workstation at very high data rates. One gigabyte (GB) of data needed to be transferred from an archiveable media on a file server to a workstation in less than 5 minutes. Equivalent size, on-line data needed to be transferred in less than 3 minutes. These requirements imply effective transfer rates on the order of four to eight megabytes per second (4-8 MB/s). The DSRS also needed to be able to send and receive data from a variety of other sources accessible from an Ethernet local area network.

Nakamoto, Glen↗

NASA NTRS1992

Data storage and retrieval system

The Data Storage and Retrieval System (DSRS) consists of off-the-shelf system components integrated as a file server supporting very large files. These files are on the order of one gigabyte of data per file, although smaller files on the order of one megabyte can be accommodated as well. For instance, one gigabyte of data occupies approximately six 9-track tape reels (recorded at 6250 bpi). Due to this large volume of media, it was desirable to 'shrink' the size of the proposed media to a single portable cassette. In addition to large size, a key requirement was that the data needs to be transferred to a (VME based) workstation at very high data rates. One gigabyte (GB) of data needed to be transferred from an archiveable media on a file server to a workstation in less than 5 minutes. Equivalent size, on-line data needed to be transferred in less than 3 minutes. These requirements imply effective transfer rates on the order of four to eight megabytes per second (4-8 MB/s). The DSRS also needed to be able to send and receive data from a variety of other sources accessible from an Ethernet local area network.

Nakamoto, Glen↗

NASA NTRS1977

Data-management and information system

User command language consists of unabbreviated English words. System allows user to create, delete, sort, merge, update, punch, or transfer all or portion of any file in system without programmer assistance.

Long, J. J.↗

NASA NTRS1991

PC-based Digital Acoustic Control System (DACS)

The PC-based Digital Acoustic Control System (DACS), which is a closed-loop system capable of precisely controlling the spectrum in real-time mode, is discussed. The system is based on integrated facility hardware including control microphones, signal conditioners, a real-time analyzer (RTA), a shaper, high capacity power amplifiers, and acoustic horns and generators. The DACS provides both an improved spectrum simulation and realtime information of pertinent test parameters that are stored in five separate files. These files can be hard copied and/or transferred to other programs to obtain a specific format of the test data. It is demonstrated that the computer interface with digital RTA and programmable filters are most effective and efficient. This facility runs independently under the control of a computer with an IEEE-488 interface to the facility hardware.

Shah, Kamlesh C.↗

DOE OSTI2019

FTLADS: Object-Logging Based Fault-Tolerant Big Data Transfer System Using Layout Aware Data Scheduling

The layout-aware data scheduling (LADS) data movement framework optimizes congestion for end-to-end data transfers. During data transfer, LADS can avoid congested storage elements by exploiting the underlying storage layout at each endpoint. This improves the I/O bandwidth and hence the data transfer rate across high-speed networks. However, the absence of fault tolerance (FT) in LADS results in data retransmission overhead and may lead to possible data integrity issues upon faults. In this paper, we propose object-logging FT mechanisms to avoid transmitting the objects that are successfully written into the parallel file system (PFS) at the sink end. Depending on the number of log files created for the whole dataset, we have classified our FT mechanisms into three different categories: file logger, transaction logger, and universal logger. Also, to address the space overhead, we have proposed different methods of populating the log files with the information of the successfully transferred objects. We have evaluated the data transfer performance and recovery time overhead of the proposed object-logging-based FT mechanisms on the LADS data transfer framework. Our experimental results reflect that FT mechanisms exhibit negligible overhead (<; 1%) with respect to the data transfer time. Yet, the fault recovery time is 10% higher than the total data transfer time at any fault point.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2003

Software for Automated Reading of STEP Files by I-DEAS(trademark)

A program called "readstep" enables the I-DEAS(tm) computer-aided-design (CAD) software to automatically read Standard for the Exchange of Product Model Data (STEP) files. (The STEP format is one of several used to transfer data between dissimilar CAD programs.) Prior to the development of "readstep," it was necessary to read STEP files into I-DEAS(tm) one at a time in a slow process that required repeated intervention by the user. In operation, "readstep" prompts the user for the location of the desired STEP files and the names of the I-DEAS(tm) project and model file, then generates an I-DEAS(tm) program file called "readstep.prg" and two Unix shell programs called "runner" and "controller." The program "runner" runs I-DEAS(tm) sessions that execute readstep.prg, while "controller" controls the execution of "runner" and edits readstep.prg if necessary. The user sets "runner" and "controller" into execution simultaneously, and then no further intervention by the user is required. When "runner" has finished, the user should see only parts from successfully read STEP files present in the model file. STEP files that could not be read successfully (e.g., because of format errors) should be regenerated before attempting to read them again.

Pinedo, John↗

DOE OSTI2020

Adaptive multi-level checkpointing

In some examples, with respect to adaptive multi-level checkpointing, a transfer parameter associated with transfer of checkpoint data from a node-local storage to a parallel file system may be ascertained for the checkpoint data stored in the node-local storage. The transfer parameter may be compared to a specified transfer parameter threshold. A determination may be made, based on the comparison of the transfer parameter to the specified transfer parameter threshold, as to whether to transfer the checkpoint data from the node-local storage to the parallel file system.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Magnetotelluric Data from Mountain Home, ID

This dataset includes magnetotelluric transfer functions in the form of EDI files for 16 stations collected by the USGS and 40 stations collected by Quantec Geoscience for Lawerence Berkeley National Lab around the Mountain Home area in Idaho. A 3D electrical resistivity model is included that images resistive and conductive bodies in the subsurface that maybe important for geothermal characterization. The model was created using ModEM using the high performance computer Yeti at the USGS.

15 GEOTHERMAL ENERGY↗

DOE OSTI2025

Data Management in the Continuum: Cross-facility Object-based Data Transfers

Scientific workflows are evolving from relying on a monolithic storage subsystem at a single High-Performance Computing (HPC) facility to using geographically distributed file systems, repositories, and cloud storage. As a result, storing, accessing, transferring, and managing scientific data have become highly complex and prone to performance inefficiencies. This paper delves into these challenges by exploring an optimized end-to-end interface designed to seamlessly connect various local and remote storage systems, enabling efficient data movement of objects across HPC–Cloud and HPC–HPC environments. We showcase this capability through an object-focused data management runtime system, discuss the effects of relaxed consistency semantics in distributed object scenarios, and illustrate its application in an earthquake simulation workflow. Besides reducing the amount of data by selectively transferring regions of interest, our facility-local results achieved a speedup of 45 × over an optimized HDF5 usage and 15 × over the HDF5 with caching by using the new interface in PDC-XF.

Bez, Jean Luca↗

DOE OSTI2019

FUSRAP Data Management and Information Transitions - 19459

The Formerly Utilized Sites Remedial Action Program (FUSRAP) was created in the mid-1970's to clean up radiological contamination resulting from early development of nuclear weapons. The US Department of Energy (US DOE) was responsible for FUSRAP until October 1997, when Congress transferred administration and execution of FUSRAP site cleanup to the US Army Corps of Engineers (USACE). Currently, the US DOE Office of Legacy Management (LM) determines the eligibility of a site for referral to USACE. USACE determines whether the site is included in FUSRAP for cleanup, conducts remedial actions, and transfers sites back to LM for long-term surveillance and maintenance (LTS and M). LM and USACE established the Data Management Working Group (DMWG) to (1) establish a common understanding of US DOE and USACE district-specific data, records practices, and requirements as they apply to FUSRAP programmatic documents for site transition and transfer; (2) establish methods to ensure accurate data transfer; and (3) develop a timeline for data and records transfer. These methods will ensure continuity of stewardship while transitioning from cleanup to LTS and M and beyond as more FUSRAP sites transfer from USACE to LM. The DMWG has met monthly and the meetings are attended by a wide range of LM and USACE project managers and subject matter experts. Discussion topics have addressed site records management, including data needs and uses; transfer and posting of Administrative Record and Permanent Record files for public accessibility; and management of legal and classified files. There have been a number of group successes, including the following: - Establishment of a common understanding of data and records practices and requirements as the sites transition and transfer from USACE to LM; - Development of effective lines of communication between USACE and LM concerning data management for transition; - Establishment of a cohesive approach to data management between the two agencies. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

NASA NTRS1992

Network issues for large mass storage requirements

File Servers and Supercomputing environments need high performance networks to balance the I/O requirements seen in today's demanding computing scenarios. UltraNet is one solution which permits both high aggregate transfer rates and high task-to-task transfer rates as demonstrated in actual tests. UltraNet provides this capability as both a Server-to-Server and Server-to-Client access network giving the supercomputing center the following advantages highest performance Transport Level connections (to 40 MBytes/sec effective rates); matches the throughput of the emerging high performance disk technologies, such as RAID, parallel head transfer devices and software striping; supports standard network and file system applications using SOCKET's based application program interface such as FTP, rcp, rdump, etc.; supports access to the Network File System (NFS) and LARGE aggregate bandwidth for large NFS usage; provides access to a distributed, hierarchical data server capability using DISCOS UniTree product; supports file server solutions available from multiple vendors, including Cray, Convex, Alliant, FPS, IBM, and others.

Perdue, James↗