Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “File transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2015

Online Data Handling and Storage at the CMS Experiment

During the LHC Long Shutdown 1, the CMS Data Acquisition (DAQ) system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and support new detector back-end electronics. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. All the metadata needed for bookkeeping are stored in files as well, in the form of small documents using the JSON encoding. The Storage and Transfer System (STS) is responsible for aggregating these files produced by the HLT, storing them temporarily and transferring them to the T0 facility at CERN for subsequent offline processing. The STS merger service aggregates the output files from the HLT from ~62 sources produced with an aggregate rate of ~2GB/s. An estimated bandwidth of 7GB/s in concurrent read/write mode is needed. Furthermore, the STS has to be able to store several days of continuous running, so an estimated of 250TB of total usable disk space is required. In this article we present the various technological and implementation choices of the three components of the STS: the distributed file system, the merger service and the transfer system.

Andre, J. M.↗
DOE OSTI2015

Portable File Format (PFF) specifications

Created at Sandia National Laboratories, the Portable File Format (PFF) allows binary data transfer across computer platforms. Although this capability is supported by many other formats, PFF files are still in use at Sandia, particularly in pulsed power research. This report provides detailed PFF specifications for accessing data without relying on legacy code.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Profiling the Usage of an Extreme-Scale Archival Storage System

Profiling the archival storage system in scientific computing environments has received much less attention compared to the parallel file system, but is equally important since it stores the final data products safely, for a long duration. In this paper, we analyze eight years worth of data transfer logs for accessing the archival file system (HPSS) in the Oak Ridge Leadership Computing Facility (OLCF), which has been hosting the world's largest supercomputers and file systems. Our analysis encompasses about 135 million data transfer activities to the 80 PB High Performance Storage System (HPSS), between 2010 and 2017. We analyze the logs from several dimensions, including studying the workload characteristics (e.g., access patterns, frequency of accesses and temporal behavior), file system characteristics (e.g., directory depth, file system scaling trends, file types), and scientific user behavior (e.g., domain-specific usage and organization). Based on the analysis, we derive insights into the future evolution of the archive in terms of provisioning, desired features and functionality from the archive software, role and right sizing of the archive tiers, quota management, and the importance of smart and efficient metadata and storage management. We believe our study will prove useful for both operating current archival storage and better provisioning future systems.

Sim, Hyogi↗
DOE OSTI2022

Moving small files in a networked environment

Globally distributed computing infrastructures, such as clouds and supercomputers, are currently used to manage data that is generated with an unprecedented speed from a variety of resources. Coping with this trend, the volume of data exchanged across distant sites increases substantially. To accelerate data transfer, high-speed networks are provided to connect remote sites. Most existing data movement solutions are optimized for moving large files. However, it is still challenging to transfer a large number of small files across networks. This disadvantage not only lowers data transfer performance, but also decreases overall system utilization. Here, we identify that moving small files is mainly constrained by degraded file system throughput, not just network performance as might be suspected. We have built a data transfer pipeline model to analyze the impact of small network I/O and storage I/O on data movement. Extending one of the widely used open source data movement solutions, GridFTP, we demonstrate several appropriate engineering approaches that mitigate the bottleneck and increase data transfer efficiency. We show optimizations that improve data transfer performance more than 5 times. In comparison to existing solutions, our approaches can save a significant amount of system resources for moving lots of small files.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

MO-B-BRD-02: 3D Printing in the Clinic

This session is designed so that the learning objectives are practical. The intent is that the attendee may take home an understanding of not just the technology, but also the logistical steps necessary to execute these 3D printing techniques in the clinic. Four practical 3D printing topics will be discussed: (i) Creating bolus and compensators for photon machines; (ii) tools for proton therapy; (iii) clinical applications in imaging; (iv) custom phantom design for clinic and research use. The use of 3D printers within the radiation oncology setting is proving to be a useful tool for creating patient specific bolus and compensators with the added benefit of cost savings. Creating the proper protocol is essential to ensuring that the desired effect is achieved and modeled in the treatment planning system. The critical choice of printer material (since it determines the interaction with the radiation) will be discussed. Selection of 3D printer type, design methods, verification of dose calculation, and the printing process will be detailed to give the basis for establishing your own protocol for electron and photon fields. A practical discussion of likely obstacles that may be encountered will be included. The diversity of systems and techniques in proton facilities leads to different facilities having very different requirements for beam modifying hardware and quality assurance devices. Many departments find the need to design and fabricate facility-specific equipment, making 3D printing an attractive technology. 3D printer applications in proton therapy will be discussed, including beam filters and compensators, and the design of proton therapy specific quality assurance tools. Quality control specific to 3D printing in proton therapy will be addressed. Advantages and disadvantages of different printing technology for these applications will also be discussed. 3D printing applications using high-resolution radiology-based imaging data will be presented. This data is used to 3D print individualized physical models of patient’s unique anatomy for aid in planning complex and challenging surgical procedures. Methods, techniques and imaging requirements for 3D printing anatomic models from imaging data will be discussed. Specific applications currently being used in the radiology clinic will be detailed. Standardized phantoms for radiation therapy are abundant. However, custom phantom designs can be advantageous for both clinical tasks and research. 3D printing is a useful method of custom fabrication that allows one to construct custom objects relatively quickly. Possibilities for custom radiotherapy phantoms range from 3D printing a hollow shell and filling the shell with tissue equivalent materials to fully printing the entire phantom with materials that are tissue equivalent as well as suitable for 3D printing. A range of materials available for use in radiotherapy phantoms and in the case of phantoms for dosimetric measurements, this choice is critical. The necessary steps required will be discussed including: modalities of 3D model generation, 3D model requirements for 3D printing, generation of machine instructions from the 3D model, and 3D printing techniques, choice of phantoms material, and troubleshooting techniques for each step in the process. Case examples of 3D printed phantoms will be shown. Learning Objectives: Understand the types of 3D modeling software required to design your device, the file formats required for data transfer from design software to 3D printer, and general troubleshooting techniques for each step of the process. Learn the differences between materials and design for photons vs. electrons vs. protons. Understand the importance of material choice and design geometries for your custom phantoms. Learn specific steps of quality assurance and quality control for 3D printed beam filters and compensators for proton therapy. Learn of special 3D printing applications for imaging. Cunha: Research support from Phillips Healthcare.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2015

MO-B-BRD-03: Principles, Pitfalls and Techniques of 3D Printing for Bolus and Compensators

This session is designed so that the learning objectives are practical. The intent is that the attendee may take home an understanding of not just the technology, but also the logistical steps necessary to execute these 3D printing techniques in the clinic. Four practical 3D printing topics will be discussed: (i) Creating bolus and compensators for photon machines; (ii) tools for proton therapy; (iii) clinical applications in imaging; (iv) custom phantom design for clinic and research use. The use of 3D printers within the radiation oncology setting is proving to be a useful tool for creating patient specific bolus and compensators with the added benefit of cost savings. Creating the proper protocol is essential to ensuring that the desired effect is achieved and modeled in the treatment planning system. The critical choice of printer material (since it determines the interaction with the radiation) will be discussed. Selection of 3D printer type, design methods, verification of dose calculation, and the printing process will be detailed to give the basis for establishing your own protocol for electron and photon fields. A practical discussion of likely obstacles that may be encountered will be included. The diversity of systems and techniques in proton facilities leads to different facilities having very different requirements for beam modifying hardware and quality assurance devices. Many departments find the need to design and fabricate facility-specific equipment, making 3D printing an attractive technology. 3D printer applications in proton therapy will be discussed, including beam filters and compensators, and the design of proton therapy specific quality assurance tools. Quality control specific to 3D printing in proton therapy will be addressed. Advantages and disadvantages of different printing technology for these applications will also be discussed. 3D printing applications using high-resolution radiology-based imaging data will be presented. This data is used to 3D print individualized physical models of patient’s unique anatomy for aid in planning complex and challenging surgical procedures. Methods, techniques and imaging requirements for 3D printing anatomic models from imaging data will be discussed. Specific applications currently being used in the radiology clinic will be detailed. Standardized phantoms for radiation therapy are abundant. However, custom phantom designs can be advantageous for both clinical tasks and research. 3D printing is a useful method of custom fabrication that allows one to construct custom objects relatively quickly. Possibilities for custom radiotherapy phantoms range from 3D printing a hollow shell and filling the shell with tissue equivalent materials to fully printing the entire phantom with materials that are tissue equivalent as well as suitable for 3D printing. A range of materials available for use in radiotherapy phantoms and in the case of phantoms for dosimetric measurements, this choice is critical. The necessary steps required will be discussed including: modalities of 3D model generation, 3D model requirements for 3D printing, generation of machine instructions from the 3D model, and 3D printing techniques, choice of phantoms material, and troubleshooting techniques for each step in the process. Case examples of 3D printed phantoms will be shown. Learning Objectives: Understand the types of 3D modeling software required to design your device, the file formats required for data transfer from design software to 3D printer, and general troubleshooting techniques for each step of the process. Learn the differences between materials and design for photons vs. electrons vs. protons. Understand the importance of material choice and design geometries for your custom phantoms. Learn specific steps of quality assurance and quality control for 3D printed beam filters and compensators for proton therapy. Learn of special 3D printing applications for imaging. Cunha: Research support from Phillips Healthcare.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2015

MO-B-BRD-04: Sterilization for 3D Printed Brachytherapy Applicators

This session is designed so that the learning objectives are practical. The intent is that the attendee may take home an understanding of not just the technology, but also the logistical steps necessary to execute these 3D printing techniques in the clinic. Four practical 3D printing topics will be discussed: (i) Creating bolus and compensators for photon machines; (ii) tools for proton therapy; (iii) clinical applications in imaging; (iv) custom phantom design for clinic and research use. The use of 3D printers within the radiation oncology setting is proving to be a useful tool for creating patient specific bolus and compensators with the added benefit of cost savings. Creating the proper protocol is essential to ensuring that the desired effect is achieved and modeled in the treatment planning system. The critical choice of printer material (since it determines the interaction with the radiation) will be discussed. Selection of 3D printer type, design methods, verification of dose calculation, and the printing process will be detailed to give the basis for establishing your own protocol for electron and photon fields. A practical discussion of likely obstacles that may be encountered will be included. The diversity of systems and techniques in proton facilities leads to different facilities having very different requirements for beam modifying hardware and quality assurance devices. Many departments find the need to design and fabricate facility-specific equipment, making 3D printing an attractive technology. 3D printer applications in proton therapy will be discussed, including beam filters and compensators, and the design of proton therapy specific quality assurance tools. Quality control specific to 3D printing in proton therapy will be addressed. Advantages and disadvantages of different printing technology for these applications will also be discussed. 3D printing applications using high-resolution radiology-based imaging data will be presented. This data is used to 3D print individualized physical models of patient’s unique anatomy for aid in planning complex and challenging surgical procedures. Methods, techniques and imaging requirements for 3D printing anatomic models from imaging data will be discussed. Specific applications currently being used in the radiology clinic will be detailed. Standardized phantoms for radiation therapy are abundant. However, custom phantom designs can be advantageous for both clinical tasks and research. 3D printing is a useful method of custom fabrication that allows one to construct custom objects relatively quickly. Possibilities for custom radiotherapy phantoms range from 3D printing a hollow shell and filling the shell with tissue equivalent materials to fully printing the entire phantom with materials that are tissue equivalent as well as suitable for 3D printing. A range of materials available for use in radiotherapy phantoms and in the case of phantoms for dosimetric measurements, this choice is critical. The necessary steps required will be discussed including: modalities of 3D model generation, 3D model requirements for 3D printing, generation of machine instructions from the 3D model, and 3D printing techniques, choice of phantoms material, and troubleshooting techniques for each step in the process. Case examples of 3D printed phantoms will be shown. Learning Objectives: Understand the types of 3D modeling software required to design your device, the file formats required for data transfer from design software to 3D printer, and general troubleshooting techniques for each step of the process. Learn the differences between materials and design for photons vs. electrons vs. protons. Understand the importance of material choice and design geometries for your custom phantoms. Learn specific steps of quality assurance and quality control for 3D printed beam filters and compensators for proton therapy. Learn of special 3D printing applications for imaging. Cunha: Research support from Phillips Healthcare.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2015

MO-B-BRD-00: Clinical Applications of 3D Printing

This session is designed so that the learning objectives are practical. The intent is that the attendee may take home an understanding of not just the technology, but also the logistical steps necessary to execute these 3D printing techniques in the clinic. Four practical 3D printing topics will be discussed: (i) Creating bolus and compensators for photon machines; (ii) tools for proton therapy; (iii) clinical applications in imaging; (iv) custom phantom design for clinic and research use. The use of 3D printers within the radiation oncology setting is proving to be a useful tool for creating patient specific bolus and compensators with the added benefit of cost savings. Creating the proper protocol is essential to ensuring that the desired effect is achieved and modeled in the treatment planning system. The critical choice of printer material (since it determines the interaction with the radiation) will be discussed. Selection of 3D printer type, design methods, verification of dose calculation, and the printing process will be detailed to give the basis for establishing your own protocol for electron and photon fields. A practical discussion of likely obstacles that may be encountered will be included. The diversity of systems and techniques in proton facilities leads to different facilities having very different requirements for beam modifying hardware and quality assurance devices. Many departments find the need to design and fabricate facility-specific equipment, making 3D printing an attractive technology. 3D printer applications in proton therapy will be discussed, including beam filters and compensators, and the design of proton therapy specific quality assurance tools. Quality control specific to 3D printing in proton therapy will be addressed. Advantages and disadvantages of different printing technology for these applications will also be discussed. 3D printing applications using high-resolution radiology-based imaging data will be presented. This data is used to 3D print individualized physical models of patient’s unique anatomy for aid in planning complex and challenging surgical procedures. Methods, techniques and imaging requirements for 3D printing anatomic models from imaging data will be discussed. Specific applications currently being used in the radiology clinic will be detailed. Standardized phantoms for radiation therapy are abundant. However, custom phantom designs can be advantageous for both clinical tasks and research. 3D printing is a useful method of custom fabrication that allows one to construct custom objects relatively quickly. Possibilities for custom radiotherapy phantoms range from 3D printing a hollow shell and filling the shell with tissue equivalent materials to fully printing the entire phantom with materials that are tissue equivalent as well as suitable for 3D printing. A range of materials available for use in radiotherapy phantoms and in the case of phantoms for dosimetric measurements, this choice is critical. The necessary steps required will be discussed including: modalities of 3D model generation, 3D model requirements for 3D printing, generation of machine instructions from the 3D model, and 3D printing techniques, choice of phantoms material, and troubleshooting techniques for each step in the process. Case examples of 3D printed phantoms will be shown. Learning Objectives: Understand the types of 3D modeling software required to design your device, the file formats required for data transfer from design software to 3D printer, and general troubleshooting techniques for each step of the process. Learn the differences between materials and design for photons vs. electrons vs. protons. Understand the importance of material choice and design geometries for your custom phantoms. Learn specific steps of quality assurance and quality control for 3D printed beam filters and compensators for proton therapy. Learn of special 3D printing applications for imaging. Cunha: Research support from Phillips Healthcare.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2015

MO-B-BRD-01: Creation of 3D Printed Phantoms for Clinical Radiation Therapy

This session is designed so that the learning objectives are practical. The intent is that the attendee may take home an understanding of not just the technology, but also the logistical steps necessary to execute these 3D printing techniques in the clinic. Four practical 3D printing topics will be discussed: (i) Creating bolus and compensators for photon machines; (ii) tools for proton therapy; (iii) clinical applications in imaging; (iv) custom phantom design for clinic and research use. The use of 3D printers within the radiation oncology setting is proving to be a useful tool for creating patient specific bolus and compensators with the added benefit of cost savings. Creating the proper protocol is essential to ensuring that the desired effect is achieved and modeled in the treatment planning system. The critical choice of printer material (since it determines the interaction with the radiation) will be discussed. Selection of 3D printer type, design methods, verification of dose calculation, and the printing process will be detailed to give the basis for establishing your own protocol for electron and photon fields. A practical discussion of likely obstacles that may be encountered will be included. The diversity of systems and techniques in proton facilities leads to different facilities having very different requirements for beam modifying hardware and quality assurance devices. Many departments find the need to design and fabricate facility-specific equipment, making 3D printing an attractive technology. 3D printer applications in proton therapy will be discussed, including beam filters and compensators, and the design of proton therapy specific quality assurance tools. Quality control specific to 3D printing in proton therapy will be addressed. Advantages and disadvantages of different printing technology for these applications will also be discussed. 3D printing applications using high-resolution radiology-based imaging data will be presented. This data is used to 3D print individualized physical models of patient’s unique anatomy for aid in planning complex and challenging surgical procedures. Methods, techniques and imaging requirements for 3D printing anatomic models from imaging data will be discussed. Specific applications currently being used in the radiology clinic will be detailed. Standardized phantoms for radiation therapy are abundant. However, custom phantom designs can be advantageous for both clinical tasks and research. 3D printing is a useful method of custom fabrication that allows one to construct custom objects relatively quickly. Possibilities for custom radiotherapy phantoms range from 3D printing a hollow shell and filling the shell with tissue equivalent materials to fully printing the entire phantom with materials that are tissue equivalent as well as suitable for 3D printing. A range of materials available for use in radiotherapy phantoms and in the case of phantoms for dosimetric measurements, this choice is critical. The necessary steps required will be discussed including: modalities of 3D model generation, 3D model requirements for 3D printing, generation of machine instructions from the 3D model, and 3D printing techniques, choice of phantoms material, and troubleshooting techniques for each step in the process. Case examples of 3D printed phantoms will be shown. Learning Objectives: Understand the types of 3D modeling software required to design your device, the file formats required for data transfer from design software to 3D printer, and general troubleshooting techniques for each step of the process. Learn the differences between materials and design for photons vs. electrons vs. protons. Understand the importance of material choice and design geometries for your custom phantoms. Learn specific steps of quality assurance and quality control for 3D printed beam filters and compensators for proton therapy. Learn of special 3D printing applications for imaging. Cunha: Research support from Phillips Healthcare.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

Fast Reactor Physics Model Verification Studies using ARC and PyARC Workflows

PyARC was recently developed at Argonne National Laboratory to automate many of the tasks required in the ARC (Argonne Reactor Computation) fast reactor simulation workflow, from input file generation, code execution, data transfer between ARC codes, and output postprocessing. PyARC will likely be the path forward to train new users of the ARC codes with the goal of wide adoption by the national laboratories, academia, and industry. In particular, for the ANL-JAEA collaboration under the Civil Nuclear Working Group (CNWG) project agreement NE-01, PyARC will be used to model the Joyo and EBR-II reactors for comparisons with measured data and calculated results from JAEA (Task 3: Fast Reactor Fuel and Core). As an additional avenue for verification and validation, this report investigates the use of PyARC towards a variety of existing ARC-based reactor models, in order to understand its efficacy in replicating the behavior of base ARC codes and better understand any limitations within modeling realistic fast reactor problems. To this end, PyARC was used to model the Joyo MKI, RBEC Benchmark-M, PRISM Mod-B, and EBR-II Run 138B cores, and its results were compared to those from existing ARC-based models. It was found that for hexagonal-based geometries PyARC was able to replicate the behavior of ARC codes to within 10 pcm for small reactor cores, and ~150pcm difference in eigenvalue for larger cores. These discrepancies are attributed primarily to differences in local mesh refinement options between ARC and PyARC, which currently cannot be resolved with PyARC’s latest version (1.6.0). In some of these cases, PyARC was used to model steady-state problems with initial core compositions originating from a prior REBUS depletion calculation. While PyARC was not designed to support such steady-state calculations, workarounds were applied to replicate the behavior of ARC-based calculations as closely as possible. Thus, these results demonstrate the wide extent to which they can be applied to fast reactor problems while still providing immense benefit to the user in terms of automating and standardizing common routines within the fast reactor analysis workflow. This study concluded that PyARC will be suitable for modeling the steady-state conditions of the EBR-II and Joyo fast reactors as part of the CNWG project agreement.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Usability and Optimization Improvements in MOOSE

The Multiphysics Object-Oriented Simulation Environment (MOOSE) framework is a foundational capability used by the Nuclear Energy Advanced Modeling and Simulation (NEAMS) program to create over 15 different simulation tools for advanced nuclear reactors. Due to MOOSE’s broad use, improvements to the framework in support of modeling and simulation goals are critical to the program. Such improvements can take many forms, including optimization, improved user experience, streamlined application programming interfaces (APIs), parallelism, and new capabilities. The work described in this report was conducted in direct support of the simulation tools and has already been deployed. The capabilities were implemented in the same order as they are covered in this report: multiple nonlinear systems in the same input file, implementation of generic field transfers and other transfer system enhancements, support for stateful material property redistribution for adaptivity and distributed meshes, and dynamic linking and loading of individually compiled applications. These four additions are fundamental capabilities that will be leveraged by many NEAMS applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Continuing Efforts on Efficiency and Performance Improvement of mdtmFTP

mdtmFTP is a high-performance data transfer tool: • Pipelined I/O-centric design to streamline data transfer • Multicore-aware data transfer middleware (MDTM) optimizes use of underlying multicore system • Extremely efficient in transferring of Lots Of Small Files • Various optimization mechanisms including Asynchronous I/O and Batch processing. • Sponsored by DOE/SC/ASCR. http://mdtm.fnal.gov $\bullet$Continuing efforts on efficiency and Performance Improvement: • Refactoring event queue mechanism • I/O optimization • Scalability evaluation

Zhang, L.↗
DOE OSTI2018

ENRESA's Spent Nuclear Fuel and Special Waste Database - 18115

Empresa Nacional de Residuos Radiactivos S.A. (ENRESA, by its initials in Spanish) Spent Nuclear Fuel (SNF) and Special Waste (SW) database development was initiated in a common effort between ENRESA and UNESA (the Spanish Association of the Electricity Industry) as a consequence of new regulatory requirements in the frame of radioactive waste management plans. Although these requirements were only mandatory for operators of nuclear power plants, ENRESA deemed necessary to have a centralized database which collected the data from the plant operators. The most relevant features of this database are: - The information collected covers not only the general characteristics of the SNF and SW but also the operation details needed for their characterization. - An application, named Gestion de Elementos Combustibles y Residuos Especiales (GECYRE, by its initials in Spanish) has been implemented for easing the queries and the generation of predefined reports. - An agile data transfer scheme, based on XML files, is used for an effective communication between ENRESA and the operators. The database is currently in production phase since 2014. During this phase it has demonstrated its effectiveness as an information repository to address the SNF and SW interim storage projects in Spain foreseen in the near future. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2022

DYnamic and Asynchronous Data Streamliner

DYAD aims to help sharing data files between producer and consumer job elements, especially between co-scheduled jobs or within an ensemble. DYAD provides the service by two components: a FLUX module and a I/O wraper set. DYAD transparently synchronizes file I/O between producer and consumer, and transfers data from the producer location to the consumer location managed by the service. Users only need to use the file path that is under the directory managed by the service.

Ahn, DongH↗
DOE OSTI2019

High-performance End-to-End Integrity Verification on Big Data Transfer

The scale of scientific data generated by experimental facilities and simulations in high-performance computing facilities has been proliferating with the emergence of IoT-based big data. In many cases, this data must be transmitted rapidly and reliably to remote facilities for storage, analysis, or sharing, for the Internet of Things (IoT) applications. Simultaneously, IoT data can be verified using a checksum after the data has been written to the disk at the destination to ensure its integrity. However, this end-to-end integrity verification inevitably creates overheads (extra disk I/O and more computation). Thus, the overall data transfer time increases. In this article, we evaluate strategies to maximize the overlap between data transfer and checksum computation for astronomical observation data. Specifically, we examine file-level and block-level (with various block sizes) pipelining to overlap data transfer and checksum computation. We analyze these pipelining approaches in the context of GridFTP, a widely used protocol for scientific data transfers. Theoretical analysis and experiments are conducted to evaluate our methods. The results show that block-level pipelining is effective in maximizing the overlap mentioned above, and can improve the overall data transfer time with end-to-end integrity verification by up to 70% compared to the sequential execution of transfer and checksum, and by up to 60% compared to file-level pipelining.

data integrity↗
DOE OSTI2015

MO-F-CAMPUS-I-01: A System for Automatically Calculating Organ and Effective Dose for Fluoroscopically-Guided Procedures

Purpose: A system was developed that automatically calculates the organ and effective dose for individual fluoroscopically-guided procedures using a log of the clinical exposure parameters. Methods: We have previously developed a dose tracking system (DTS) to provide a real-time color-coded 3D- mapping of skin dose. This software produces a log file of all geometry and exposure parameters for every x-ray pulse during a procedure. The data in the log files is input into PCXMC, a Monte Carlo program that calculates organ and effective dose for projections and exposure parameters set by the user. We developed a MATLAB program to read data from the log files produced by the DTS and to automatically generate the definition files in the format used by PCXMC. The processing is done at the end of a procedure after all exposures are completed. Since there are thousands of exposure pulses with various parameters for fluoroscopy, DA and DSA and at various projections, the data for exposures with similar parameters is grouped prior to entry into PCXMC to reduce the number of Monte Carlo calculations that need to be performed. Results: The software developed automatically transfers data from the DTS log file to PCXMC and runs the program for each grouping of exposure pulses. When the dose from all exposure events are calculated, the doses for each organ and all effective doses are summed to obtain procedure totals. For a complicated interventional procedure, the calculations can be completed on a PC without manual intervention in less than 30 minutes depending on the level of data grouping. Conclusion: This system allows organ dose to be calculated for individual procedures for every patient without tedious calculations or data entry so that estimates of stochastic risk can be obtained in addition to the deterministic risk estimate provided by the DTS. Partial support from NIH grant R01EB002873 and Toshiba Medical Systems Corp.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

GPU Direct I/O with HDF5

Exascale HPC systems are being designed with accelerators, such as GPUs, to accelerate parts of applications. In machine learning workloads as well as large-scale simulations that use GPUs as accelerators, the CPU (or host) memory is currently used as a buffer for data transfers between GPU (or device) memory and the file system. If the CPU does not need to operate on the data, then this is sub-optimal because it wastes host memory by reserving space for duplicated data. Furthermore, this “bounce buffer” approach wastes CPU cycles spent on transferring data. A new technique, NVIDIA GPUDirect Storage (GDS), can eliminate the need to use the host memory as a bounce buffer. Thereby, it becomes possible to transfer data directly between the device memory and the file system. This direct data path shortens latency by omitting the extra copy and enables higher-bandwidth. To take full advantage of GDS in existing applications, it is necessary to provide support with existing I/O libraries, such as HDF5 and MPI-IO, which are heavily used in applications. In this paper, we describe our effort of integrating GDS with HDF5, the top I/O library at NERSC and at DOE leadership computing facilities. We design and implement this integration using a HDF5 Virtual File Driver (VFD). The GDS VFD provides a file system abstraction to the application that allows HDF5 applications to perform I/O without the need to move data between CPUs and GPUs explicitly. We compare performance of the HDF5 GDS VFD with explicit data movement approaches and demonstrate superior performance with the GDS method.

Ravi, J↗
DOE OSTI2016

Optimizing End-to-End Big Data Transfers over Terabits Network Infrastructure

While future terabit networks hold the promise of significantly improving big-data motion among geographically distributed data centers, significant challenges must be overcome even on today's 100 gigabit networks to realize end-to-end performance. Multiple bottlenecks exist along the end-to-end path from source to sink, for instance, the data storage infrastructure at both the source and sink and its interplay with the wide-area network are increasingly the bottleneck to achieving high performance. In this study, we identify the issues that lead to congestion on the path of an end-to-end data transfer in the terabit network environment, and we present a new bulk data movement framework for terabit networks, called LADS. LADS exploits the underlying storage layout at each endpoint to maximize throughput without negatively impacting the performance of shared storage resources for other users. LADS also uses the Common Communication Interface (CCI) in lieu of the sockets interface to benefit from hardware-level zero-copy, and operating system bypass capabilities when available. It can further improve data transfer performance under congestion on the end systems using buffering at the source using flash storage. With our evaluations, we show that LADS can avoid congested storage elements within the shared storage resource, improving input/output bandwidth, and data transfer rates across the high speed networks. We also investigate the performance degradation problems of LADS due to I/O contention on the parallel file system (PFS), when multiple LADS tools share the PFS. We design and evaluate a meta-scheduler to coordinate multiple I/O streams while sharing the PFS, to minimize the I/O contention on the PFS. Finally, with our evaluations, we observe that LADS with meta-scheduling can further improve the performance by up to 14 percent relative to LADS without meta-scheduling.

97 MATHEMATICS AND COMPUTING↗