Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “file system”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32
All sourcesNASA NTRSDOE OSTI
NASA NTRS2007

Creating Interactive Graphical Overlays in the Advanced Weather Interactive Processing System (AWIPS) Using Shapefiles and DGM Files

Graphical overlays can be created in real-time in the Advanced Weather Interactive Processing System (AWIPS) using shapefiles or DARE Graphics Metafile (DGM) files. This presentation describes how to create graphical overlays on-the-fly for AWIPS, by using two examples of AWIPS applications that were created by the Applied Meteorology Unit (AMU). The first example is the Anvil Threat Corridor Forecast Tool, which produces a shapefile that depicts a graphical threat corridor of the forecast movement of thunderstorm anvil clouds, based on the observed or forecast upper-level winds. This tool is used by the Spaceflight Meteorology Group (SMG) and 45th Weather Squadron (45 WS) to analyze the threat of natural or space vehicle-triggered lightning over a location. The second example is a launch and landing trajectory tool that produces a DGM file that plots the ground track of space vehicles during launch or landing. The trajectory tool can be used by SMG and the 45 WS forecasters to analyze weather radar imagery along a launch or landing trajectory. Advantages of both file types will be listed.

Barrett, Joe H., III↗
NASA NTRS1994

Architecture and evolution of Goddard Space Flight Center Distributed Active Archive Center

The Goddard Space Flight Center (GSFC) Distributed Active Archive Center (DAAC) has been developed to enhance Earth Science research by improved access to remote sensor earth science data. Building and operating an archive, even one of a moderate size (a few Terabytes), is a challenging task. One of the critical components of this system is Unitree, the Hierarchical File Storage Management System. Unitree, selected two years ago as the best available solution, requires constant system administrative support. It is not always suitable as an archive and distribution data center, and has moderate performance. The Data Archive and Distribution System (DADS) software developed to monitor, manage, and automate the ingestion, archive, and distribution functions turned out to be more challenging than anticipated. Having the software and tools is not sufficient to succeed. Human interaction within the system must be fully understood to improve efficiency to improve efficiency and ensure that the right tools are developed. One of the lessons learned is that the operability, reliability, and performance aspects should be thoroughly addressed in the initial design. However, the GSFC DAAC has demonstrated that it is capable of distributing over 40 GB per day. A backup system to archive a second copy of all data ingested is under development. This backup system will be used not only for disaster recovery but will also replace the main archive when it is unavailable during maintenance or hardware replacement. The GSFC DAAC has put a strong emphasis on quality at all level of its organization. A Quality team has also been formed to identify quality issues and to propose improvements. The DAAC has conducted numerous tests to benchmark the performance of the system. These tests proved to be extremely useful in identifying bottlenecks and deficiencies in operational procedures.

Bedet, Jean-Jacques↗
NASA NTRS2015

MERRA-2: File Specification

The second Modern-Era Retrospective analysis for Research and Applications (MERRA-2) is a NASA atmospheric reanalysis that begins in 1980. It replaces the original MERRA reanalysis (Rienecker et al., 2011) using an upgraded version of the Goddard Earth Observing System Model, Version 5 (GEOS-5) data assimilation system. The file collections for MERRA-2 are described in detail in this document, including some important changes from those of the MERRA dataset (Lucchesi, 2012).

MERRA-2↗
DOE OSTI2012

File I/O for MPI Applications in Redundant Execution Scenarios

As multi-petascale and exa-scale high-performance computing (HPC) systems inevitably have to deal with a number of resilience challenges, such as a significant growth in component count and smaller circuit sizes with lower circuit voltages, redundancy may offer an acceptable level of resilience that traditional fault tolerance techniques, such as checkpoint/restart, do not. Although redundancy in HPC is quite controversial due to the associated cost for redundant components, the constantly increasing number of cores-per-processor is tilting this cost calculation toward a system design where computation, such as for redundancy, is much cheaper and communication, needed for checkpoint/restart, is much more expensive. Recent research and development activities in redundancy for Message Passing Interface (MPI) applications focused on availability/reliability models and replication algorithms. This paper takes a first step toward solving an open research problem associated with running a parallel application redundantly, which is file I/O under redundancy. The approach intercepts file I/O calls made by a redundant application to employ coordination protocols that execute file I/O operations in a redundancy-oblivious fashion when accessing a node-local file system, or in a redundancy-aware fashion when accessing a shared networked file system. A proof-of concept prototype is presented and a number of coordination protocols are described and evaluated. The results show the performance impact for redundantly accessing a shared networked file system, but also demonstrate the capability to regain performance by utilizing MPI communication between replicas and parallel file I/O.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2014

Performance model-directed data sieving for high-performance I/O

Many scientific computing applications and engineering simulations exhibit noncontiguous I/O access patterns. Data sieving is an important technique to improve the performance of noncontiguous I/O accesses by combining small and noncontiguous requests into a large and contiguous request. It has been proven effective even though more data are potentially accessed than demanded. In this study, we propose a new data sieving approach namely performance model-directed data sieving, or PMD data sieving in short. It improves the existing data sieving approach from two aspects: (1) dynamically determines when it is beneficial to perform data sieving; and (2) dynamically determines how to perform data sieving if beneficial. It improves the performance of the existing data sieving approach considerably and reduces the memory consumption as verified by both theoretical analysis and experimental results. Given the importance of supporting noncontiguous accesses effectively and reducing the memory pressure in a large-scale system, the proposed PMD data sieving approach in this research holds a great promise and will have an impact on high-performance I/O systems.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1974

A catalog system for remote-sensing data

The Practical System for Cataloging, Indexing, and Retrieval of Remote Sensing Data developed by the Interdisciplinary Remote Sensing Group at the University of Wisconsin consists of a card catalog, a site-index-map, a site-index-file, an industry-index-file, and a project-index-file. The system is designed for retrieval of remote-sensing data which include imagery, magnetic tapes, flight logs, maps, ground-truth reports, and research reports containing raw data. It can be operated by conventional library methods, but provision has been made for digitizing the system for computer retrieval.

Singh, R. S.↗
DOE OSTI2018

I/O load balancing for big data HPC applications

High Performance Computing (HPC) big data problems require efficient distributed storage systems. However, at scale, such storage systems often experience load imbalance and resource contention due to two factors: the bursty nature of scientific application I/O; and the complex I/O path that is without centralized arbitration and control. For example, the extant Lustre parallel file system-that supports many HPC centers-comprises numerous components connected via custom network topologies, and serves varying demands of a large number of users and applications. Consequently, some storage servers can be more loaded than others, which creates bottlenecks and reduces overall application I/O performance. Existing solutions typically focus on per application load balancing, and thus are not as effective given their lack of a global view of the system. In this paper, we propose a data-driven approach to load balance the I/O servers at scale, targeted at Lustre deployments. To this end, we design a global mapper on Lustre Metadata Server, which gathers runtime statistics from key storage components on the I/O path, and applies Markov chain modeling and a minimum-cost maximum-flow algorithm to decide where data should be placed. Evaluation using a realistic system simulator and a real setup shows that our approach yields better load balancing, which in turn can improve end-to-end performance.

Paul, Arnab K.↗
NASA NTRS2013

ILRS Station Reporting

Network stations provided system configuration documentation upon joining the ILRS. This information, found in the various site and system log files available on the ILRS website, is essential to the ILRS analysis centers, combination centers, and general user community. Therefore, it is imperative that the station personnel inform the ILRS community in a timely fashion when changes to the system occur. This poster provides some information about the various documentation that must be maintained. The ILRS network consists of over fifty global sites actively ranging to over sixty satellites as well as five lunar reflectors. Information about these stations are available on the ILRS website (http://ilrs.gsfc.nasa.gov/network/stations/index.html). The ILRS Analysis Centers must have current information about the stations and their system configuration in order to use their data in generation of derived products. However, not all information available on the ILRS website is as up-to-date as necessary for correct analysis of their data.

satellites↗
NASA NTRS1983

Comparative analysis of data base management systems

A study to determine if the Remote File Inquiry (RFI) system would handle the future requirements of the user community is discussed. RFI is a locally written and locally maintained on-line query/update package. The current and future on-line requirements of the user community were studied. Additional consideration was given to the types of data structuring the users required. The survey indicated the features of greatest benefit were: sort, subtotals, totals, record selection, storage of queries, global updating and the ability to page break. The major deficiencies were: one level of hierarchy, excessive response time, software unreliability, difficult to add, delete and modify records, complicated error messages and the lack of ability to perform interfield comparisons. Missing features users required were: formatted screens, interfield comparions, interfield arithmetic, multiple file access, security and data integrity. The survey team recommended Kennedy Space Center move forward to state-of-the-art software, a Data Base Management System which is thoroughly tested and easy to implement and use.

Smith, R.↗
DOE OSTI2020

ESGF Future Architecture Report (V.1.1)

This document reports the findings of the Earth System Grid Federation Future Architecture Face-to-Face Meeting held in the UK 5-7 November 2019, attended by technical representatives from the ESGF partner organisations. ESGF is an International federated archive that provides access to Earth science data. It is fundamentally a distributed file management system with its main function being data dissemination through the provision of services for efficient replication and search. It has been in operation for over ten years and has undergone continuous improvements and enhancements over this time span. The Future Architecture is a major review of the software system including a fundamental reassessment of its requirements with a view to develop a new overall system architecture. This has been initiated in the light of challenges around the maintenance of the existing software legacy and a changing technology landscape and practices in the communities it serves. The work is being carried out under the oversight of the ESGF Executive Committee1,2 and is coordinated activity between ESGF and the European IS-ENES3 project.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2013

Life Cycle Analysis Data and Results for Geothermal and Other Electricity Generation Technologies

Life cycle analysis (LCA) is an environmental assessment method that quantifies the environmental performance of a product system over its entire lifetime, from cradle to grave. Based on a set of relevant metrics, the method is aptly suited for comparing the environmental performance of competing products systems. This file contains LCA data and results for electric power production including geothermal power. The LCA for electric power has been broken down into two life cycle stages, namely plant and fuel cycles. Relevant metrics include the energy ratio and greenhouse gas (GHG) ratios, where the former is the ratio of system input energy to total lifetime electrical energy out and the latter is the ratio of the sum of all incurred greenhouse gases (in CO2 equivalents) divided by the same energy output. Specific information included herein are material to power (MPR) ratios for a range of power technologies for conventional thermoelectric, renewables (including three geothermal power technologies), and coproduced natural gas/geothermal power. For the geothermal power scenarios, the MPRs include the casing, cement, diesel, and water requirements for drilling wells and topside piping. Also included herein are energy and GHG ratios for plant and fuel cycle stages for the range of considered electricity generating technologies. Some of this information are MPR data extracted directly from the literature or from models (eg. ICARUS - a subset of ASPEN models) and others (energy and GHG ratios) are results calculated using GREET models and MPR data. MPR data for wells included herein were based on the Argonne well materials model and GETEM well count results.

15 GEOTHERMAL ENERGY↗
DOE OSTI2026

Generalizable Web User Interface for Scalable and Streamlined Deployment of Building Energy Management Systems in Small and Medium-Sized Commercial Buildings

Small and medium-sized commercial buildings (SMCBs) comprise 94% of US commercial buildings yet face significant barriers to implementing building energy management systems despite advances in smart device technology. Existing solutions present critical limitations: cloud-based API solutions simplify deployment but create vendor lock-in constraints; commercial integrated software solutions ensure compatibility via standardized protocols but require substantial cost and technical expertise; open-source IoT platforms offer cost-effective vendor independence but provide insufficient standardized protocol support for commercial building automation. This research presents a generalizable web user interface framework that bridges the gap between evolving smart device capabilities and lagging software infrastructure for SMCBs. The proposed system integrates VOLTTRON open-source middleware with an automated configuration converter that transforms unified specifications written in YAML, a human-readable data-serialization format, into system-specific files, streamlining manual setup processes. The vendor-agnostic architecture supports industry-standard protocols (BACnet and Modbus) and semantic building models while providing adaptive web interfaces that dynamically adjust to various building configurations. Demonstrations through simulation-based testing and a field deployment show automatic interface adaptation across heterogeneous HVAC systems and multizone monitoring. The automated configuration converter also substantially reduces labor-intensive setup.

Chung, Jihoon [ORNL] (ORCID:0000000184880815)↗
DOE OSTI2025

DEDUPKV: A Space-Efficient and High-Performance Key-Value Store via Fine-Grained Deduplication

Log-Structured Merge Tree (LSM-tree) based key-value stores excel in write-intensive environments but suffer from data duplication, consuming up to 49% of storage space in LSM-tree-based key-value store deployments. Traditional solutions like compression and coarse-grained file system-level deduplication introduce overhead or have limited effectiveness. In this study, we propose DedupKV, a fine-grained deduplication framework tailored for LSM-tree, maximizing data reduction efficiency while minimizing write stalls and read overheads. DedupKV features three key innovations: (1) FLUSH-integrated inline deduplication, which removes duplicates during memory-to-storage writes; (2) WAL file-based offline deduplication, repurposing write-ahead logs to avoid double writes; and (3) elastic execution, dynamically balancing inline and offline deduplication based on memory pressure and workload intensity. Additionally, dynamic granularity management reduces deduplication metadata overhead. We implemented these four ideas in RocksDB for the first time and conducted experiments in a Linux environment. Our evaluation shows that WAL file-based offline deduplication and DedupKV outperform BlobDB by 33% and 23%, respectively, in write-heavy workloads, while reducing write amplification by 1.2 ×, 2 ×, and 1.6 × for real KV datasets.

Jamil, Safdar [Sogang University]↗
NASA NTRS1974

An overview of the National Space Science data Center Standard Information Retrieval System (SIRS)

A general overview is given of the National Space Science Data Center (NSSDC) Standard Information Retrieval System. A description, in general terms, the information system that contains the data files and the software system that processes and manipulates the files maintained at the Data Center. Emphasis is placed on providing users with an overview of the capabilities and uses of the NSSDC Standard Information Retrieval System (SIRS). Examples given are taken from the files at the Data Center. Detailed information about NSSDC data files is documented in a set of File Users Guides, with one user's guide prepared for each file processed by SIRS. Detailed information about SIRS is presented in the SIRS Users Guide.

Shapiro, A.↗
DOE OSTI2013

Taming parallel I/O complexity with auto-tuning

We present an auto-tuning system for optimizing I/O performance of HDF5 applications and demonstrate its value across platforms, applications, and at scale. The system uses a genetic algorithm to search a large space of tunable parameters and to identify effective settings at all layers of the parallel I/O stack. The parameter settings are applied transparently by the auto-tuning system via dynamically intercepted HDF5 calls. To validate our auto-tuning system, we applied it to three I/O benchmarks (VPIC, VORPAL, and GCRM) that replicate the I/O activity of their respective applications. We tested the system with different weak-scaling configurations (128, 2048, and 4096 CPU cores) that generate 30 GB to 1 TB of data, and executed these configurations on diverse HPC platforms (Cray XE6, IBM BG/P, and Dell Cluster). In all cases, the auto-tuning framework identified tunable parameters that substantially improved write performance over default system settings. In conclusion, we consistently demonstrate I/O write speedups between 2x and 100x for test configurations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2013

A Temporal Locality-Aware Page-Mapped Flash Translation Layer

The poor performance of random writes has been a cause of major concern which needs to be addressed to better utilize the potential of flash in enterprise-scale environments. We examine one of the important causes of this poor performance: the design of the flash translation layer (FTL) which performs the virtual-to-physical address translations and hides the erase-before-write characteristics of flash. We propose a complete paradigm shift in the design of the core FTL engine from the existing techniques with our Demand-Based Flash Translation Layer (DFTL) which selectively caches page- level address mappings. Our experimental evaluation using FlashSim with realistic enterprise-scale workloads endorses the utility of DFTL in enterprise-scale storage systems by demonstrating: 1) improved performance, 2) reduced garbage collection overhead and 3) better overload behavior compared with hybrid FTL schemes which are the most popular implementation methods. For example, a predominantly random-write dominant I/O trace from an OLTP application running at a large financial institution shows a 78% improvement in average response time (due to a 3-fold reduction in operations of the garbage collector), compared with the hybrid FTL scheme. Even for the well-known read-dominant TPC-H benchmark, for which DFTL introduces additional overheads, we improve system response time by 56%. Moreover, interestingly, when write-back cache on DFTL-based SSD is enabled, DFTL even outperforms the page-based FTL scheme, improving their response time by 72% in Financial trace.

Address Mapping Table↗
DOE OSTI2015

LADS: Optimizing Data Transfers using Layout-Aware Data Scheduling

While future terabit networks hold the promise of signifi- cantly improving big-data motion among geographically distributed data centers, significant challenges must be overcome even on today s 100 gigabit networks to real- ize end-to-end performance. Multiple bottlenecks exist along the end-to-end path from source to sink. Data stor- age infrastructure at both the source and sink and its in- terplay with the wide-area network are increasingly the bottleneck to achieving high performance. In this paper, we identify the issues that lead to congestion on the path of an end-to-end data transfer in the terabit network en- vironment, and we present a new bulk data movement framework called LADS for terabit networks. LADS ex- ploits the underlying storage layout at each endpoint to maximize throughput without negatively impacting the performance of shared storage resources for other users. LADS also uses the Common Communication Interface (CCI) in lieu of the sockets interface to use zero-copy, OS-bypass hardware when available. It can further im- prove data transfer performance under congestion on the end systems using buffering at the source using flash storage. With our evaluations, we show that LADS can avoid congested storage elements within the shared stor- age resource, improving I/O bandwidth, and data transfer rates across the high speed networks.

Data Management↗