SEARCH · Engineering Papers

Results for “File Systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26

All sources NASA NTRS DOE OSTI

NASA NTRS1984

Mathematical synthesization of complex structures

A mathematical synthesization approach for rapidly modeling and analyzing complex structures comprised of hundreds or thousands of repeating individual structural members, interconnecting hardware, and other components was described. The techniques are applicable to any structure with repeating structural members. The capabilities are embodied within the Interactive Design and Evaluation of Advanced Spacecraft (IDEAS) computer-aided design system. IDEAS has a full range of integrated spacecraft modeling, design, analysis, performance, and cost estimating capabilities which include about 40 technical program and executive, data base, and file management systems. With IDEAS, a single user at an interactive terminal can create, design, analyze, and conduct parametric studies of Earth orbiting spacecraft in a timely, cost-efficient manner. The system is particularly useful in the conceptual design phases of advanced space missions when a multiplicity of concepts must be evaluated. The IDEAS interactive finite-element modeling capabilities are discussed. The mathematical synthesization programs were structured not only to provide a detailed finite-element model of the structure but also to automatically perform the many auxiliary computations needed to analyze the spacecraft in its orbital operational environment.

Garrett, L. B.↗

DOE OSTI2025

Regen: An object layout regenerator on large-scale production HPC systems

This article proposes an object layout regenerator called Regen which regenerates and removes the object layout dynamically to improve the read performance of applications. Regen first detects frequent access patterns from the I/O requests of the applications. Second, Regen reorganizes the objects and regenerates or preallocates new object layouts according to the identified access patterns. Finally, Regen removes or reuses the obsolete or regenerated object layouts as necessary. As a result, Regen accelerates access to objects by providing a flexible object layout. We implement Regen as a framework on top of Proactive Data Container (PDC) and evaluate it on Cori supercomputer, a production-scale HPC system, by using realistic HPC I/O benchmarks. The experimental results show that Regen improves the I/O performance by up to 16.92 × compared with an existing system.

Distributed file system↗

NASA NTRS2007

Creating Interactive Graphical Overlays in the Advanced Weather Interactive Processing System (AWIPS) Using Shapefiles and DGM Files

Graphical overlays can be created in real-time in the Advanced Weather Interactive Processing System (AWIPS) using shapefiles or DARE Graphics Metafile (DGM) files. This presentation describes how to create graphical overlays on-the-fly for AWIPS, by using two examples of AWIPS applications that were created by the Applied Meteorology Unit (AMU). The first example is the Anvil Threat Corridor Forecast Tool, which produces a shapefile that depicts a graphical threat corridor of the forecast movement of thunderstorm anvil clouds, based on the observed or forecast upper-level winds. This tool is used by the Spaceflight Meteorology Group (SMG) and 45th Weather Squadron (45 WS) to analyze the threat of natural or space vehicle-triggered lightning over a location. The second example is a launch and landing trajectory tool that produces a DGM file that plots the ground track of space vehicles during launch or landing. The trajectory tool can be used by SMG and the 45 WS forecasters to analyze weather radar imagery along a launch or landing trajectory. Advantages of both file types will be listed.

Barrett, Joe H., III↗

NASA NTRS1994

Architecture and evolution of Goddard Space Flight Center Distributed Active Archive Center

The Goddard Space Flight Center (GSFC) Distributed Active Archive Center (DAAC) has been developed to enhance Earth Science research by improved access to remote sensor earth science data. Building and operating an archive, even one of a moderate size (a few Terabytes), is a challenging task. One of the critical components of this system is Unitree, the Hierarchical File Storage Management System. Unitree, selected two years ago as the best available solution, requires constant system administrative support. It is not always suitable as an archive and distribution data center, and has moderate performance. The Data Archive and Distribution System (DADS) software developed to monitor, manage, and automate the ingestion, archive, and distribution functions turned out to be more challenging than anticipated. Having the software and tools is not sufficient to succeed. Human interaction within the system must be fully understood to improve efficiency to improve efficiency and ensure that the right tools are developed. One of the lessons learned is that the operability, reliability, and performance aspects should be thoroughly addressed in the initial design. However, the GSFC DAAC has demonstrated that it is capable of distributing over 40 GB per day. A backup system to archive a second copy of all data ingested is under development. This backup system will be used not only for disaster recovery but will also replace the main archive when it is unavailable during maintenance or hardware replacement. The GSFC DAAC has put a strong emphasis on quality at all level of its organization. A Quality team has also been formed to identify quality issues and to propose improvements. The DAAC has conducted numerous tests to benchmark the performance of the system. These tests proved to be extremely useful in identifying bottlenecks and deficiencies in operational procedures.

Bedet, Jean-Jacques↗

NASA NTRS2015

MERRA-2: File Specification

The second Modern-Era Retrospective analysis for Research and Applications (MERRA-2) is a NASA atmospheric reanalysis that begins in 1980. It replaces the original MERRA reanalysis (Rienecker et al., 2011) using an upgraded version of the Goddard Earth Observing System Model, Version 5 (GEOS-5) data assimilation system. The file collections for MERRA-2 are described in detail in this document, including some important changes from those of the MERRA dataset (Lucchesi, 2012).

MERRA-2↗

DOE OSTI2023

Design and Implementation of Burst Buffer Over-Subscription Scheme for HPC Storage Systems

Not Available

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1974

A catalog system for remote-sensing data

The Practical System for Cataloging, Indexing, and Retrieval of Remote Sensing Data developed by the Interdisciplinary Remote Sensing Group at the University of Wisconsin consists of a card catalog, a site-index-map, a site-index-file, an industry-index-file, and a project-index-file. The system is designed for retrieval of remote-sensing data which include imagery, magnetic tapes, flight logs, maps, ground-truth reports, and research reports containing raw data. It can be operated by conventional library methods, but provision has been made for digitizing the system for computer retrieval.

Singh, R. S.↗

DOE OSTI2018

I/O load balancing for big data HPC applications

High Performance Computing (HPC) big data problems require efficient distributed storage systems. However, at scale, such storage systems often experience load imbalance and resource contention due to two factors: the bursty nature of scientific application I/O; and the complex I/O path that is without centralized arbitration and control. For example, the extant Lustre parallel file system-that supports many HPC centers-comprises numerous components connected via custom network topologies, and serves varying demands of a large number of users and applications. Consequently, some storage servers can be more loaded than others, which creates bottlenecks and reduces overall application I/O performance. Existing solutions typically focus on per application load balancing, and thus are not as effective given their lack of a global view of the system. In this paper, we propose a data-driven approach to load balance the I/O servers at scale, targeted at Lustre deployments. To this end, we design a global mapper on Lustre Metadata Server, which gathers runtime statistics from key storage components on the I/O path, and applies Markov chain modeling and a minimum-cost maximum-flow algorithm to decide where data should be placed. Evaluation using a realistic system simulator and a real setup shows that our approach yields better load balancing, which in turn can improve end-to-end performance.

Paul, Arnab K.↗

NASA NTRS2013

ILRS Station Reporting

Network stations provided system configuration documentation upon joining the ILRS. This information, found in the various site and system log files available on the ILRS website, is essential to the ILRS analysis centers, combination centers, and general user community. Therefore, it is imperative that the station personnel inform the ILRS community in a timely fashion when changes to the system occur. This poster provides some information about the various documentation that must be maintained. The ILRS network consists of over fifty global sites actively ranging to over sixty satellites as well as five lunar reflectors. Information about these stations are available on the ILRS website (http://ilrs.gsfc.nasa.gov/network/stations/index.html). The ILRS Analysis Centers must have current information about the stations and their system configuration in order to use their data in generation of derived products. However, not all information available on the ILRS website is as up-to-date as necessary for correct analysis of their data.

satellites↗

NASA NTRS1983

Comparative analysis of data base management systems

A study to determine if the Remote File Inquiry (RFI) system would handle the future requirements of the user community is discussed. RFI is a locally written and locally maintained on-line query/update package. The current and future on-line requirements of the user community were studied. Additional consideration was given to the types of data structuring the users required. The survey indicated the features of greatest benefit were: sort, subtotals, totals, record selection, storage of queries, global updating and the ability to page break. The major deficiencies were: one level of hierarchy, excessive response time, software unreliability, difficult to add, delete and modify records, complicated error messages and the lack of ability to perform interfield comparisons. Missing features users required were: formatted screens, interfield comparions, interfield arithmetic, multiple file access, security and data integrity. The survey team recommended Kennedy Space Center move forward to state-of-the-art software, a Data Base Management System which is thoroughly tested and easy to implement and use.

Smith, R.↗

DOE OSTI2020

ESGF Future Architecture Report (V.1.1)

This document reports the findings of the Earth System Grid Federation Future Architecture Face-to-Face Meeting held in the UK 5-7 November 2019, attended by technical representatives from the ESGF partner organisations. ESGF is an International federated archive that provides access to Earth science data. It is fundamentally a distributed file management system with its main function being data dissemination through the provision of services for efficient replication and search. It has been in operation for over ten years and has undergone continuous improvements and enhancements over this time span. The Future Architecture is a major review of the software system including a fundamental reassessment of its requirements with a view to develop a new overall system architecture. This has been initiated in the light of challenges around the maintenance of the existing software legacy and a changing technology landscape and practices in the communities it serves. The work is being carried out under the oversight of the ESGF Executive Committee1,2 and is coordinated activity between ESGF and the European IS-ENES3 project.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Generalizable Web User Interface for Scalable and Streamlined Deployment of Building Energy Management Systems in Small and Medium-Sized Commercial Buildings

Small and medium-sized commercial buildings (SMCBs) comprise 94% of US commercial buildings yet face significant barriers to implementing building energy management systems despite advances in smart device technology. Existing solutions present critical limitations: cloud-based API solutions simplify deployment but create vendor lock-in constraints; commercial integrated software solutions ensure compatibility via standardized protocols but require substantial cost and technical expertise; open-source IoT platforms offer cost-effective vendor independence but provide insufficient standardized protocol support for commercial building automation. This research presents a generalizable web user interface framework that bridges the gap between evolving smart device capabilities and lagging software infrastructure for SMCBs. The proposed system integrates VOLTTRON open-source middleware with an automated configuration converter that transforms unified specifications written in YAML, a human-readable data-serialization format, into system-specific files, streamlining manual setup processes. The vendor-agnostic architecture supports industry-standard protocols (BACnet and Modbus) and semantic building models while providing adaptive web interfaces that dynamically adjust to various building configurations. Demonstrations through simulation-based testing and a field deployment show automatic interface adaptation across heterogeneous HVAC systems and multizone monitoring. The automated configuration converter also substantially reduces labor-intensive setup.

Chung, Jihoon [ORNL] (ORCID:0000000184880815)↗

DOE OSTI2025

DEDUPKV: A Space-Efficient and High-Performance Key-Value Store via Fine-Grained Deduplication

Log-Structured Merge Tree (LSM-tree) based key-value stores excel in write-intensive environments but suffer from data duplication, consuming up to 49% of storage space in LSM-tree-based key-value store deployments. Traditional solutions like compression and coarse-grained file system-level deduplication introduce overhead or have limited effectiveness. In this study, we propose DedupKV, a fine-grained deduplication framework tailored for LSM-tree, maximizing data reduction efficiency while minimizing write stalls and read overheads. DedupKV features three key innovations: (1) FLUSH-integrated inline deduplication, which removes duplicates during memory-to-storage writes; (2) WAL file-based offline deduplication, repurposing write-ahead logs to avoid double writes; and (3) elastic execution, dynamically balancing inline and offline deduplication based on memory pressure and workload intensity. Additionally, dynamic granularity management reduces deduplication metadata overhead. We implemented these four ideas in RocksDB for the first time and conducted experiments in a Linux environment. Our evaluation shows that WAL file-based offline deduplication and DedupKV outperform BlobDB by 33% and 23%, respectively, in write-heavy workloads, while reducing write amplification by 1.2 ×, 2 ×, and 1.6 × for real KV datasets.

Jamil, Safdar [Sogang University]↗

NASA NTRS1974

An overview of the National Space Science data Center Standard Information Retrieval System (SIRS)

A general overview is given of the National Space Science Data Center (NSSDC) Standard Information Retrieval System. A description, in general terms, the information system that contains the data files and the software system that processes and manipulates the files maintained at the Data Center. Emphasis is placed on providing users with an overview of the capabilities and uses of the NSSDC Standard Information Retrieval System (SIRS). Examples given are taken from the files at the Data Center. Detailed information about NSSDC data files is documented in a set of File Users Guides, with one user's guide prepared for each file processed by SIRS. Detailed information about SIRS is presented in the SIRS Users Guide.

Shapiro, A.↗

DOE OSTI2015

LADS: Optimizing Data Transfers using Layout-Aware Data Scheduling

While future terabit networks hold the promise of signifi- cantly improving big-data motion among geographically distributed data centers, significant challenges must be overcome even on today s 100 gigabit networks to real- ize end-to-end performance. Multiple bottlenecks exist along the end-to-end path from source to sink. Data stor- age infrastructure at both the source and sink and its in- terplay with the wide-area network are increasingly the bottleneck to achieving high performance. In this paper, we identify the issues that lead to congestion on the path of an end-to-end data transfer in the terabit network en- vironment, and we present a new bulk data movement framework called LADS for terabit networks. LADS ex- ploits the underlying storage layout at each endpoint to maximize throughput without negatively impacting the performance of shared storage resources for other users. LADS also uses the Common Communication Interface (CCI) in lieu of the sockets interface to use zero-copy, OS-bypass hardware when available. It can further im- prove data transfer performance under congestion on the end systems using buffering at the source using flash storage. With our evaluations, we show that LADS can avoid congested storage elements within the shared stor- age resource, improving I/O bandwidth, and data transfer rates across the high speed networks.

Data Management↗

NASA NTRS1973

NASIS data base management system: IBM 360 TSS implementation. Volume 6: NASIS message file

The message file for the NASA Aerospace Safety Information System (NASIS) is discussed. The message file contains all the message and term explanations for the system. The data contained in the file can be broken down into three separate sections: (1) global terms, (2) local terms, and (3) system messages. The various terms are defined and their use within the system is explained.

Source record↗

NASA NTRS1973

NASIS data base management system - IBM 360/370 OS MVT implementation. 6: NASIS message file

Source record↗

DOE OSTI2016

Data Movement in Data-Intensive High Performance Computing.

The cost of executing a floating point operation has been decreasing for decades at a much higher rate than that of moving data. Bandwidth and latency, two key metrics that determine the cost of moving data, have degraded significantly relative to processor cycle time and execution rate. Despite the limitation of sub-micron processor technology and the end of Dennard scaling, this trend will continue in the short-term making data movement a performance-limiting factor and an energy/power efficiency concern. Even more so in the context of large-scale and data-intensive systems and workloads. This chapter gives an overview of the aspects of moving data across a system, from the storage system to the computing system down to the node and processor level, with case study and contributions from researchers at the San Diego Supercomputer Center, the Oak Ridge National Laboratory, the Pacific Northwest National Laboratory, and the University of Delaware.

Data Movement↗