Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “FILES”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
NASA NTRS1991

Solar-A reformatted data files and observing log

An overview is presented of the Solar-A telemetry data files which are to be created and the format and organization which the files are to use. The organization chosen is to be efficient in space, to facilitate access to the data, and to allow the data to be transportable to different machines. An observing log file is to be created automatically, using the reformatted data files as the input. It will be possible to perform searches with the observing log to list cases where instruments are in certain modes and/or seeing certain signal levels. A user will be able to search the observing log and obtain a list of all cases where a given set of conditions are satisfied. An event log will be created listing the times when the instrument or spacecraft modes change.

Morrison, M. D.↗
NASA NTRS1993

RAMA: A file system for massively parallel computers

This paper describes a file system design for massively parallel computers which makes very efficient use of a few disks per processor. This overcomes the traditional I/O bottleneck of massively parallel machines by storing the data on disks within the high-speed interconnection network. In addition, the file system, called RAMA, requires little inter-node synchronization, removing another common bottleneck in parallel processor file systems. Support for a large tertiary storage system can easily be integrated in lo the file system; in fact, RAMA runs most efficiently when tertiary storage is used.

Miller, Ethan L.↗
NASA NTRS1994

Dynamic file-access characteristics of a production parallel scientific workload

Multiprocessors have permitted astounding increases in computational performance, but many cannot meet the intense I/O requirements of some scientific applications. An important component of any solution to this I/O bottleneck is a parallel file system that can provide high-bandwidth access to tremendous amounts of data in parallel to hundreds or thousands of processors. Most successful systems are based on a solid understanding of the expected workload, but thus far there have been no comprehensive workload characterizations of multiprocessor file systems. This paper presents the results of a three week tracing study in which all file-related activity on a massively parallel computer was recorded. Our instrumentation differs from previous efforts in that it collects information about every I/O request and about the mix of jobs running in a production environment. We also present the results of a trace-driven caching simulation and recommendations for designers of multiprocessor file systems.

Kotz, David↗
NASA NTRS1995

File-access characteristics of parallel scientific workloads

Phenomenal improvements in the computational performance of multiprocessors have not been matched by comparable gains in I/O system performance. This imbalance has resulted in I/O becoming a significant bottleneck for many scientific applications. One key to overcoming this bottleneck is improving the performance of parallel file systems. The design of a high-performance parallel file system requires a comprehensive understanding of the expected workload. Unfortunately, until recently, no general workload studies of parallel file systems have been conducted. The goal of the CHARISMA project was to remedy this problem by characterizing the behavior of several production workloads, on different machines, at the level of individual reads and writes. The first set of results from the CHARISMA project describe the workloads observed on an Intel iPSC/860 and a Thinking Machines CM-5. This paper is intended to compare and contrast these two workloads for an understanding of their essential similarities and differences, isolating common trends and platform-dependent variances. Using this comparison, we are able to gain more insight into the general principles that should guide parallel file-system design.

Nieuwejaar, Nils↗
NASA NTRS1996

The Global File System

The global file system (GFS) is a prototype design for a distributed file system in which cluster nodes physically share storage devices connected via a network-like fiber channel. Networks and network-attached storage devices have advanced to a level of performance and extensibility so that the previous disadvantages of shared disk architectures are no longer valid. This shared storage architecture attempts to exploit the sophistication of storage device technologies whereas a server architecture diminishes a device's role to that of a simple component. GFS distributes the file system responsibilities across processing nodes, storage across the devices, and file system resources across the entire storage pool. GFS caches data on the storage devices instead of the main memories of the machines. Consistency is established by using a locking mechanism maintained by the storage devices to facilitate atomic read-modify-write operations. The locking mechanism is being prototyped in the Silicon Graphics IRIX operating system and is accessed using standard Unix commands and modules.

Soltis, Steven R.↗
NASA NTRS1996

Understanding Customer Dissatisfaction with Underutilized Distributed File Servers

An important trend in the design of storage subsystems is a move toward direct network attachment. Network-attached storage offers the opportunity to off-load distributed file system functionality from dedicated file server machines and execute many requests directly at the storage devices. For this strategy to lead to better performance, as perceived by users, the response time of distributed operations must improve. In this paper we analyze measurements of an Andrew file system (AFS) server that we recently upgraded in an effort to improve client performance in our laboratory. While the original server's overall utilization was only about 3%, we show how burst loads were sufficiently intense to lead to period of poor response time significant enough to trigger customer dissatisfaction. In particular, we show how, after adjusting for network load and traffic to non-project servers, 50% of the variation in client response time was explained by variation in server central processing unit (CPU) use. That is, clients saw long response times in large part because the server was often over-utilized when it was used at all. Using these measures, we see that off-loading file server work in a network-attached storage architecture has to potential to benefit user response time. Computational power in such a system scales directly with storage capacity, so the slowdown during burst period should be reduced.

Riedel, Erik↗
NASA NTRS1996

Optimizing Input/Output Using Adaptive File System Policies

Parallel input/output characterization studies and experiments with flexible resource management algorithms indicate that adaptivity is crucial to file system performance. In this paper we propose an automatic technique for selecting and refining file system policies based on application access patterns and execution environment. An automatic classification framework allows the file system to select appropriate caching and pre-fetching policies, while performance sensors provide feedback used to tune policy parameters for specific system environments. To illustrate the potential performance improvements possible using adaptive file system policies, we present results from experiments involving classification-based and performance-based steering.

Madhyastha, Tara M.↗
NASA NTRS1995

File-System Workload on a Scientific Multiprocessor

Many scientific applications have intense computational and I/O requirements. Although multiprocessors have permitted astounding increases in computational performance, the formidable I/O needs of these applications cannot be met by current multiprocessors a their I/O subsystems. To prevent I/O subsystems from forever bottlenecking multiprocessors and limiting the range of feasible applications, new I/O subsystems must be designed. The successful design of computer systems (both hardware and software) depends on a thorough understanding of their intended use. A system designer optimizes the policies and mechanisms for the cases expected to most common in the user's workload. In the case of multiprocessor file systems, however, designers have been forced to build file systems based only on speculation about how they would be used, extrapolating from file-system characterizations of general-purpose workloads on uniprocessor and distributed systems or scientific workloads on vector supercomputers (see sidebar on related work). To help these system designers, in June 1993 we began the Charisma Project, so named because the project sought to characterize 1/0 in scientific multiprocessor applications from a variety of production parallel computing platforms and sites. The Charisma project is unique in recording individual read and write requests-in live, multiprogramming, parallel workloads (rather than from selected or nonparallel applications). In this article, we present the first results from the project: a characterization of the file-system workload an iPSC/860 multiprocessor running production, parallel scientific applications at NASA's Ames Research Center.

Kotz, David↗
NASA NTRS2004

Automated File Transfer and Storage Management Concepts for Space

This presentation will summarize work that has been done to prototype and analyze approaches for automated file transfer and storage management for space missions. The concepts were prototyped in an environment with data files being generated at the target mission rates and stored in onboard files. The space-to-ground link was implemented using a channel simulator to introduce representative mission delays and errors. The system was operated for days with data files building up on the spacecraft and periodically being transferred to ground storage during a limited contact time. Overall performance was measured to identify limits under which the entire data volume could be transferred automatically while still fitting into the mission s limited contact time. The overall concepts, measurements, and results will be presented.

Hogie, Keith↗
NASA NTRS2004

[PVFS 2000: An operational parallel file system for Beowulf]

The approach has been to develop Parallel Virtual File System version 2 (PVFS2) , retaining the basic philosophy of the original file system but completely rewriting the code. It shows the architecture of the server and client components. BMI - BMI is the network abstraction layer. It is designed with a common driver and modules for each protocol supported. The interface is non-blocking, and provides mechanisms for optimizations including pinning user buffers. Currently TCP/IP and GM(Myrinet) modules have been implemented. Trove -Trove is the storage abstraction layer. It provides for storing both data spaces and name/value pairs. Trove can also be implemented using different underlying storage mechanisms including native files, raw disk partitions, SQL and other databases. The current implementation uses native files for data spaces and Berkeley db for name/value pairs.

Ligon, Walt↗
NASA NTRS2010

Tropospheric Emission Spectrometer Product File Readers

TES Product File Reader software extracts data from publicly available Tropospheric Emission Spectrometer (TES) HDF (Hierarchical Data Format) product data files using publicly available format specifications for scientific analysis in IDL (interactive data language). In this innovation, the software returns data fields as simple arrays for a given file. A file name is provided, and the contents are returned as simple IDL variables.

Fisher, Brendan M.↗
NASA NTRS2006

Shuttle Data Center File-Processing Tool in Java

A Java-language computer program has been written to facilitate mining of data in files in the Shuttle Data Center (SDC) archives. This program can be executed on a variety of workstations or via Web-browser programs. This program is partly similar to prior C-language programs used for the same purpose, while differing from those programs in that it exploits the platform-neutrality of Java in implementing several features that are important for analysis of large sets of time-series data. The program supports regular expression queries of SDC archive files, reads the files, interleaves the time-stamped samples according to a chosen output, then transforms the results into that format. A user can choose among a variety of output file formats that are useful for diverse purposes, including plotting, Markov modeling, multivariate density estimation, and wavelet multiresolution analysis, as well as for playback of data in support of simulation and testing.

Barry, Matthew R.↗
NASA NTRS2014

File Specification for the 7-km GEOS-5 Nature Run, Ganymed Release Non-Hydrostatic 7-km Global Mesoscale Simulation

This document describes the gridded output files produced by a two-year global, non-hydrostatic mesoscale simulation for the period 2005-2006 produced with the non-hydrostatic version of GEOS-5 Atmospheric Global Climate Model (AGCM). In addition to standard meteorological parameters (wind, temperature, moisture, surface pressure), this simulation includes 15 aerosol tracers (dust, sea-salt, sulfate, black and organic carbon), O3, CO and CO2. This model simulation is driven by prescribed sea-surface temperature and sea-ice, daily volcanic and biomass burning emissions, as well as high-resolution inventories of anthropogenic sources. A description of the GEOS-5 model configuration used for this simulation can be found in Putman et al. (2014). The simulation is performed at a horizontal resolution of 7 km using a cubed-sphere horizontal grid with 72 vertical levels, extending up to to 0.01 hPa (approximately 80 km). For user convenience, all data products are generated on two logically rectangular longitude-latitude grids: a full-resolution 0.0625 deg grid that approximately matches the native cubed-sphere resolution, and another 0.5 deg reduced-resolution grid. The majority of the full-resolution data products are instantaneous with some fields being time-averaged. The reduced-resolution datasets are mostly time-averaged, with some fields being instantaneous. Hourly data intervals are used for the reduced-resolution datasets, while 30-minute intervals are used for the full-resolution products. All full-resolution output is on the model's native 72-layer hybrid sigma-pressure vertical grid, while the reduced-resolution output is given on native vertical levels and on 48 pressure surfaces extending up to 0.02 hPa. Section 4 presents additional details on horizontal and vertical grids. Information of the model surface representation can be found in Appendix B. The GEOS-5 product is organized into file collections that are described in detail in Appendix C. Additional details about variables listed in this file specification can be found in a separate document, the GEOS-5 File Specification Variable Definition Glossary. Documentation about the current access methods for products described in this document can be found on the GEOS-5 Nature Run portal: http://gmao.gsfc.nasa.gov/projects/G5NR. Information on the scientific quality of this simulation will appear in a forthcoming NASA Technical Report Series on Global Modeling and Data Assimilation to be available from http://gmao.gsfc.nasa.gov/pubs/tm/.

GEOS-5↗
NASA NTRS2018

File Specification for M2AMIP Products: Version Number - 1.0

The Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2) is an atmospheric reanalysis computed with the Goddard Earth Observing (EOS) System, Version 5.12.4 (GEOS) data assimilation system (Gelaro et al., 2017). To supplement the reanalysis, the GEOS General Circulation Model (GCM) used in MERRA-2 has been used to generate a 10-member ensemble of simulations, configured following the convention of the Atmospheric Model Intercomparison Project (AMIP; Gates et al., 1992). Each ensemble member was initialized using meteorological fields from a different date in November 1979. The AMIP simulations used the sea-surface temperature (SST) and sea-ice boundary conditions that were used in MERRA-2 (Bosilovich et al., 2016). This 10-member ensemble of AMIP simulations, denoted M2AMIP, is available for download in a group of self-describing files, which are documented in this office note. All data collections are provided on the same horizontal grid as MERRA-2. This grid has 576 points in the longitudinal direction and 361 points in the latitudinal direction, corresponding to a resolution of 0.625 degrees by 0.5 degrees. Although data collections are available at this grid, all fields are computed on a cubed-sphere grid with an approximate resolution of 50 km by 50 km and are then spatially interpolated to the latitude-longitude grid. There are no changes in the vertical grids used: variables are provided on either the native vertical grid of 72 model layers, or interpolated to 42 standard pressure levels. Unlike MERRA, no data collections are available at the vertical layer edges. More details on the grid are provided in Section 4. MERRA-2 introduced observation-based precipitation forcing for the land surface parameterization and the corresponding variable PRECTOTCORR in the MERRA-2 FLX (surface turbulent fluxes and related quantities) and LFO (land-surface forcing) collections (see Section 6; Reichle et al., 2017). While this variable is still available for M2AMIP, there was no observation-based forcing, making the value identical to the model derived precipitation, PRECTOT. Similarly, without data assimilation, the values for the analysis increments, D*DTANA, in the tendency and vertically integrated file collections are zero. The M2AMIP data are available for download online through the NASA Center for Climate Simulation (NCCS) DataPortal (https://portal.nccs.nasa.gov/datashare/gmao_m2amip/). Data are arranged in subdirectories based on ensemble member, followed by year and month. Control files that are compatible with the Grid Analysis and Display System (GrADS) are available in the ctl_daily and ctl_monthly directories for the hourly, three hourly, and monthly mean data. Control files for the monthly mean diurnal cycle can be found in the ctl_diurnal subdirectory within the directory for each individual ensemble member.

M2AMIP↗
DOE OSTI2012

A Next-Generation Parallel File System Environment for the OLCF

When deployed in 2008/2009 the Spider system at the Oak Ridge National Laboratory s Leadership Computing Facility (OLCF) was the world s largest scale Lustre parallel file system. Envisioned as a shared parallel file system capable of delivering both the bandwidth and capacity requirements of the OLCF s diverse computational environment, Spider has since become a blueprint for shared Lustre environments deployed worldwide. Designed to support the parallel I/O requirements of the Jaguar XT5 system and other smallerscale platforms at the OLCF, the upgrade to the Titan XK6 heterogeneous system will begin to push the limits of Spider s original design by mid 2013. With a doubling in total system memory and a 10x increase in FLOPS, Titan will require both higher bandwidth and larger total capacity. Our goal is to provide a 4x increase in total I/O bandwidth from over 240GB=sec today to 1TB=sec and a doubling in total capacity. While aggregate bandwidth and total capacity remain important capabilities, an equally important goal in our efforts is dramatically increasing metadata performance, currently the Achilles heel of parallel file systems at leadership. We present in this paper an analysis of our current I/O workloads, our operational experiences with the Spider parallel file systems, the high-level design of our Spider upgrade, and our efforts in developing benchmarks that synthesize our performance requirements based on our workload characterization studies.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2015

Cooperative storage of shared files in a parallel computing system with dynamic block size

Improved techniques are provided for parallel writing of data to a shared object in a parallel computing system. A method is provided for storing data generated by a plurality of parallel processes to a shared object in a parallel computing system. The method is performed by at least one of the processes and comprises: dynamically determining a block size for storing the data; exchanging a determined amount of the data with at least one additional process to achieve a block of the data having the dynamically determined block size; and writing the block of the data having the dynamically determined block size to a file system. The determined block size comprises, e.g., a total amount of the data to be stored divided by the number of parallel processes. The file system comprises, for example, a log structured virtual parallel file system, such as a Parallel Log-Structured File System (PLFS).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

pcircle - A Suite of Scalable Parallel File System Tools

Most of the software related to file system are written for conventional local file system, they are serialized and can't take advantage of the benefit of a large scale parallel file system. "pcircle" software builds on top of ubiquitous MPI in cluster computing environment and "work-stealing" pattern to provide a scalable, high-performance suite of file system tools. In particular - it implemented parallel data copy and parallel data checksumming, with advanced features such as async progress report, checkpoint and restart, as well as integrity checking.

WANG, FEIYI↗
DOE OSTI2021

Grand unified file indexing

Systems and methods are disclosed for a unified file index for a file system. In one example, a Grand Unified File Index (GUFI) includes a tree replicating the directory hierarchy of one or more primary filesystems, and individual metadata stores for each directory. The GUFI tree permits fast traversal, efficient user space access controls, and simple tree directed operations such as renames, moves, or permission changes. In some examples, the individual metadata stores can be implemented as embedded databases on flash storage for speed. In some examples, use of summary tables at the directory or subtree level can eliminate wasteful executions, prune tree traversal, and further improve performance. In various examples, efficient operation can be achieved from laptop to supercomputer scale, across a wide mix of file distributions and filesystems.

97 MATHEMATICS AND COMPUTING↗