SEARCH · Engineering Papers

Results for “File System”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 667 records · Page 37

All sources NASA NTRS DOE OSTI

DOE OSTI2020

PDQ Users Manual Version 1 for PDQ Code Version 1.12

PDQ is a tool for the management of the input and execution of batch jobs for simulation codes that use a text based input system. It accomplishes this goal by operating at two levels. First, it takes input file templates (commonly known at LANL as input deck templates) and creates multiple instantiations by performing substitutions of data from table files into symbols (variables) found in the template. Second, it provides commands to submit the created files to the SLURM batch system for execution. These two activities taken together produce a whole that is greater than the sum of its parts and provides an elegant way of executing studies across multiple similar simulations while minimizing the risk of typographical errors in the input files.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2010

Planetary Image Geometry Library

The Planetary Image Geometry (PIG) library is a multi-mission library used for projecting images (EDRs, or Experiment Data Records) and managing their geometry for in-situ missions. A collection of models describes cameras and their articulation, allowing application programs such as mosaickers, terrain generators, and pointing correction tools to be written in a multi-mission manner, without any knowledge of parameters specific to the supported missions. Camera model objects allow transformation of image coordinates to and from view vectors in XYZ space. Pointing models, specific to each mission, describe how to orient the camera models based on telemetry or other information. Surface models describe the surface in general terms. Coordinate system objects manage the various coordinate systems involved in most missions. File objects manage access to metadata (labels, including telemetry information) in the input EDRs and RDRs (Reduced Data Records). Label models manage metadata information in output files. Site objects keep track of different locations where the spacecraft might be at a given time. Radiometry models allow correction of radiometry for an image. Mission objects contain basic mission parameters. Pointing adjustment ("nav") files allow pointing to be corrected. The object-oriented structure (C++) makes it easy to subclass just the pieces of the library that are truly mission-specific. Typically, this involves just the pointing model and coordinate systems, and parts of the file model. Once the library was developed (initially for Mars Polar Lander, MPL), adding new missions ranged from two days to a few months, resulting in significant cost savings as compared to rewriting all the application programs for each mission. Currently supported missions include Mars Pathfinder (MPF), MPL, Mars Exploration Rover (MER), Phoenix, and Mars Science Lab (MSL). Applications based on this library create the majority of operational image RDRs for those missions. A Java wrapper around the library allows parts of it to be used from Java code (via a native JNI interface). Future conversions of all or part of the library to Java are contemplated.

Deen, Robert C.↗

DOE OSTI2019

Data transfer between scientific facilities – bottleneck analysis, insights, and optimizations

Wide area file transfers play an important role in many science applications. File transfer tools typically deliver the highest performance for datasets with a small number of large files, but many science datasets consist of many small files. Thus it is important to understand the factors that contribute to the decrease in wide area data transfer performance for datasets with many small files. To this end, we (i) benchmark the performance of subsystems involved in end-to-end file transfer between two HPC facilities for a many-file dataset that is representative of production science transfers; (ii) characterize the per-file overhead introduced by different subsystems; (iii) identify potential dependencies and bottlenecks; (iv) study the effectiveness of transferring many files concurrently as a means of reducing per-file overheads; and (v) prototype a prefetching mechanism as an alternative of concurrency to reduce the per-file overhead on source storage system. We show that both concurrency and prefetching can help reduce the per-file overhead significantly. A reasonable level of concurrency combined with prefetching can bring the per-file overhead down to a negligible level.

Liu, Yuanlai↗

DOE OSTI2025

RE-INTEGRATE EMT Simulation Tool: Input Data Processing Layer for Bulk Power System

This paper introduces an advanced input data processing layer for EMT simulations of large-scale bulk power systems. The paper proposes two versions of the RE-INTEGRATE EMT simulation tool, RE-INTEGRATE Gen-0 and RE-INTEGRATE Gen-1, which are developed to enhance simulation generalizability, scalability, and accuracy. The framework leverages a generic class design for components to incorporate linear equations, which are generated by discretizing the Differential-Algebraic Equations (DAEs) that represent the dynamics of the components. In addition, the framework employs a parsing algorithm that parses a power system’s raw and dyr files to generate a connectivity graph which is then traversed to form the overall system’s dynamics. The proposed input data processing layer is used to simulate the IEEE 39-bus test system. The obtained results demonstrate the framework’s capability to achieve simulation scalability and accuracy. Further, the results indicate that EMT simulations performed using the proposed automations can effectively handle complex grid configurations.

Mishra, Rahul [ORNL] (ORCID:0000000328205932)↗

NASA NTRS1995

NASA-IGES Translator and Viewer

NASA-IGES Translator (NIGEStranslator) is a batch program that translates a general IGES (Initial Graphics Exchange Specification) file to a NASA-IGES-Nurbs-Only (NINO) file. IGES is the most popular geometry exchange standard among Computer Aided Geometric Design (CAD) systems. NINO format is a subset of IGES, implementing the simple and yet the most popular NURBS (Non-Uniform Rational B-Splines) representation. NIGEStranslator converts a complex IGES file to the simpler NINO file to simplify the tasks of CFD grid generation for models in CAD format. The NASA-IGES Viewer (NIGESview) is an Open-Inventor-based, highly interactive viewer/ editor for NINO files. Geometry in the IGES files can be viewed, copied, transformed, deleted, and inquired. Users can use NIGEStranslator to translate IGES files from CAD systems to NINO files. The geometry then can be examined with NIGESview. Extraneous geometries can be interactively removed, and the cleaned model can be written to an IGES file, ready to be used in grid generation.

Chou, Jin J.↗

NASA NTRS1999

Next Generation Space Telescope Integrated Science Module Data System

The Data system for the Next Generation Space Telescope (NGST) Integrated Science Module (ISIM) is the primary data interface between the spacecraft, telescope, and science instrument systems. This poster includes block diagrams of the ISIM data system and its components derived during the pre-phase A Yardstick feasibility study. The poster details the hardware and software components used to acquire and process science data for the Yardstick instrument compliment, and depicts the baseline external interfaces to science instruments and other systems. This baseline data system is a fully redundant, high performance computing system. Each redundant computer contains three 150 MHz power PC processors. All processors execute a commercially available real time multi-tasking operating system supporting, preemptive multi-tasking, file management and network interfaces. These six processors in the system are networked together. The spacecraft interface baseline is an extension of the network, which links the six processors. The final selection for Processor busses, processor chips, network interfaces, and high-speed data interfaces will be made during mid 2002.

Schnurr, Richard G.↗

NASA NTRS2005

BEAM Technology Flight Demonstration

As technologies advance, their growing complexity makes them harder to maintain. Detection methods for isolating and identifying impending problems are needed to balance this complexity. Through comparison of signal pairs from onboard sensors, the Beacon-based Exception Analysis For Multimissions (BEAM) algorithm can identify and help classify deviations in system operation from a data-trained statistical model. The goal of this task is to mature BEAM and validate its performance on a flying test bed. A series of F-18 flight demonstrations with BEAM monitoring engine parameters in real time was used to demonstrate in-the-field readiness. Captured F-18 and simulated F-18 engine data were used in model creation and training. The algorithm was then ported to the embedded system with a data buffering, file writing, and data-time-stamp monitoring shell to reduce the impact of embedded system faults on BEAM'S ability to correctly identify engine faults. Embedded system testing identified hardware related restrictions and contributed to iterative improvements in the code's runtime performance. The system was flown with forced engine flameouts and other pilot induced faults to simulate operation out of the norm. Successful detection of these faults, confirmed through post-flight data analysis, helped BEAM achieve TRL6.

Wang, David↗

NASA NTRS2007

Solving Equations of Multibody Dynamics

Darts++ is a computer program for solving the equations of motion of a multibody system or of a multibody model of a dynamic system. It is intended especially for use in dynamical simulations performed in designing and analyzing, and developing software for the control of, complex mechanical systems. Darts++ is based on the Spatial-Operator- Algebra formulation for multibody dynamics. This software reads a description of a multibody system from a model data file, then constructs and implements an efficient algorithm that solves the dynamical equations of the system. The efficiency and, hence, the computational speed is sufficient to make Darts++ suitable for use in realtime closed-loop simulations. Darts++ features an object-oriented software architecture that enables reconfiguration of system topology at run time; in contrast, in related prior software, system topology is fixed during initialization. Darts++ provides an interface to scripting languages, including Tcl and Python, that enable the user to configure and interact with simulation objects at run time.

Jain, Abhinandan↗

NASA NTRS1982

An automated data management/analysis system for space shuttle orbiter tiles

An engineering data management system was combined with a nonlinear stress analysis program to provide a capability for analyzing a large number of tiles on the space shuttle orbiter. Tile geometry data and all data necessary of define the tile loads environment accessed automatically as needed for the analysis of a particular tile or a set of tiles. User documentation provided includes: (1) description of computer programs and data files contained in the system; (2) definitions of all engineering data stored in the data base; (3) characteristics of the tile anaytical model; (4) instructions for preparation of user input; and (5) a sample problem to illustrate use of the system. Description of data, computer programs, and analytical models of the tile are sufficiently detailed to guide extension of the system to include additional zones of tiles and/or additional types of analyses

Giles, G. L.↗

NASA NTRSDate not supplied

High Performance Access to Archival Data Stored in HDF4 and HDF5 on Cloud Object Stores Without Reformatting the Files

Cloud computing offers numerous advantages for users of extensive Earth science data collections. These benefits encompass direct online access to data files and granules from any location, scalable access supporting parallel computing workflows, and flexible computing tools enabling innovative experimentation with processing techniques. However, older archival file formats designed for distinct computing systems hinder efficient access to decade-long time-series data when compared to data stored in modern cloud-optimized formats like Web Object Stores (WOS), exemplified by Amazon Web Services’ Simple Storage Service (S3). We describe DMR++ (Dataset Metadata Response plus plus), a technology facilitating efficient access to HDF5 (Hierarchical Data Format, version 5) and HDF4 files stored on WOS systems without requiring data reformatting. DMR++ achieves performance comparable to technologies like Zarr while preserving the original file structure, a substantial benefit considering the vast quantity of archival files held by organizations such as NASA. Moreover, DMR++ typically outperforms cloud-optimized versions of HDF5. Essentially an XML (Extensible Markup Language) document usually stored alongside the described data, DMR++ can also be generated on-the-fly but is generally created during data staging to the WOS. Archival files that use HDF4/5 often store large arrays of numerical data. The data in these files is often compressed, typically reducing their size by a factor of four or more. To achieve efficient access to portions of those arrays, they are 'chunked' into smaller sub-arrays, each individually compressed. The chunk size is a compromise, where spinning disks can efficiently access data in smaller chunks while S3 favors larger chunks. A simple optimization of aggregating smaller chunks that are stored adjacently, transferring them in a single access and then individually decompressing them will improve performance. NASA data pose an additional challenge: special Application Programmer Interface (API) libraries are often needed to compute some variables. These libraries are incompatible with WOS environments. Our solution involves storing computed values in the DMR++ document or a companion file, making them accessible like other variables and eliminating the need for specialized APIs. We outline specific optimizations for both satellite grid and swath data stored in HDF4-EOS2 (Earth Observing System).

James Gallagher↗

NASA NTRS1996

Development of Secondary Archive System at Goddard Space Flight Center Version 0 Distributed Active Archive Center

The Goddard Space Flight Center (GSFC) version 0 Distributed Active Archive Center (DAAC) has been developed to support existing and pre Earth Observing System (EOS) Earth science datasets, facilitate the scientific research, and test EOS data and information system (EOSDIS) concepts. To ensure that no data is ever lost, each product received at GSFC DAAC is archived on two different media, VHS and digital linear tape (DLT). The first copy is made on VHS tape and is under the control of UniTree. The second and third copies are made to DLT and VHS media under a custom built software package named 'Archer'. While Archer provides only a subset of the functions available with commercial software like UniTree, it supports migration between near-line and off-line media and offers much greater performance and flexibility to satisfy the specific needs of a data center. Archer is specifically designed to maximize total system throughput, rather than focusing on the turn-around time for individual files. The commercial off the shelf software (COTS) hierarchical storage management (HSM) products evaluated were mainly concerned with transparent, interactive, file access to the end-user, rather than a batch-orientated, optimizable (based on known data file characteristics) data archive and retrieval system. This is critical to the distribution requirements of the GSFC DAAC where orders for 5000 or more files at a time are received. Archer has the ability to queue many thousands of file requests and to sort these requests into internal processing schedules that optimize overall throughput. Specifically, mount and dismount, tape load and unload cycles, and tape motion are minimized. This feature did not seem to be available in many COTS pacages. Archer also uses a generic tar tape format that allows tapes to be read by many different systems rather than the proprietary format found in most COTS packages. This paper discusses some of the specific requirements at GSFC DAAC, the motivations for implementing the Archer system, and presents a discussion of the Archer design that resulted.

Sherman, Mark↗

DOE OSTI2016

An asynchronous traversal engine for graph-based rich metadata management

Rich metadata in high-performance computing (HPC) systems contains extended information about users, jobs, data files, and their relationships. Property graphs are a promising data model to represent heterogeneous rich metadata flexibly. Specifically, a property graph can use vertices to represent different entities and edges to record the relationships between vertices with unique annotations. The high-volume HPC use case, with millions of entities and relationships, naturally requires an out-of-core distributed property graph database, which must support live updates (to ingest production information in real time), low-latency point queries (for frequent metadata operations such as permission checking), and large-scale traversals (for provenance data mining). Among these needs, large-scale property graph traversals are particularly challenging for distributed graph storage systems. Most existing graph systems implement a "level synchronous" breadth-first search algorithm that relies on global synchronization in each traversal step. This performs well in many problem domains; but a rich metadata management system is characterized by imbalanced graphs, long traversal lengths, and concurrent workloads, each of which has the potential to introduce or exacerbate stragglers (i.e., abnormally slow steps or servers in a graph traversal) that lead to low overall throughput for synchronous traversal algorithms. Previous research indicated that the straggler problem can be mitigated by using asynchronous traversal algorithms, and many graph-processing frameworks have successfully demonstrated this approach. Such systems require the graph to be loaded into a separate batch-processing framework instead of being iteratively accessed, however. In this work, we investigate a general asynchronous graph traversal engine that can operate atop a rich metadata graph in its native format. We outline a traversal-aware query language and key optimizations (traversal-affiliate caching and execution merging) necessary for efficient performance. We further explore the effect of different graph partitioning strategies on the traversal performance for both synchronous and asynchronous traversal engines. Our experiments show that the asynchronous graph traversal engine is more efficient than its synchronous counterpart in the case of HPC rich metadata processing, where more servers are involved and larger traversals are needed. Furthermore, the asynchronous traversal engine is more adaptive to different graph partitioning strategies.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2019

Open Data, Models, and Codes for Vanadium Redox Batch Cell Systems: A Systems Approach Using Zero-Dimensional Models

Abstract In this paper, we study, analyze, and validate some important zero-dimensional physics-based models for vanadium redox batch cell (VRBC) systems and formulate an adequate physics-based model that can predict the battery performance accurately. In the model formulation process, a systems approach to multiple parameters estimation has been conducted using VRBC systems at low C-rates (~C/30). In this batch cell system, the effect of ions' crossover through the membrane is dominant, and therefore, the capacity loss phenomena can be explicitly observed. Paradoxically, this means that using the batch system might be a better approach for identifying a more suitable model describing the effect of ions transport. Next, we propose an efficient systems approach, which enables to help understand the battery performance quickly by estimating all parameters of the battery system. Finally, open source codes, executable files, and experimental data are provided to enable people's access to robust and accurate models and optimizers. In battery simulations, different models and optimizers describing the same systems produce different values of the estimated parameters. Providing an open access platform can accelerate the process to arrive at robust models and optimizers by continuous modification from the users' side.

25 ENERGY STORAGE↗

DOE OSTI2023

Recent streamflow trends across permafrost basins of North America: Datasets

Climate change impacts, including changing temperatures, precipitation, and vegetation, are widely anticipated to cause major shifts to the permafrost with resulting impacts to hydro-ecosystems across the high latitudes of the globe. However, it is challenging to examine streamflow shifts in these regions owing to a paucity of data, discontinuity of records, and other issues related to data consistency and accuracy. We looked at recent changes in streamflow over 1976-2021 in watersheds affected by varying degrees of permafrost coverage to characterize trends and drivers for a range of watersheds across North America. Data sets are described in detail in the paper associated with this data set, Bennett et al. 2023, Front. Water - Water and Critical Zone, DOI: 10.3389/frwa.2023.1099660.These data contain CSV files of the streamflow, climate, and land surface characteristics for several sites located across the high latitude regions of North America. Both observed and reanalysis data products are provided. These files can be opened using Excel or a text editor, or they can be read, and analyzed in software tools such as Python or R. A brief description of the files is below, and more details can be found in the Methods section.rabpro_stats_north_select_74_55m.csv - This file describes the observed gages used in the analysis.GF31_23_metadat.csv - This file describes the 23 permafrost systems. Columns are as described in rabpro_stats_north_select_74_55m.csv above, with rabpro_id, the id used for the timeseries file mapping in GF31_23_time_series.csv.GF31_23_time_series.csv - This file contains the time series data for the stations described in GF31_23_metadat.csv.GF31_random_reaches_1583.csv - This file describes the 1583 randomly selected permafrost-dominant sites for machine learning analysis.era5_GF31_monthly_vars_random_reaches_1583.csv - This file contains the monthly ERA5 land data for the 1583 randomly selected permafrost-dominant sites.observed.zip: USGS and Hydat station data for the 74 gages analyzed in this study. Monthly, seasonal, and annual streamflow observations for minimum streamflow, mean streamflow, and maximum streamflow. Units are m3/sec. 1975-2022.streamflow_daily_GF31_infilled_1979_2022.csv - infilled daily streamflow data (infilled using GloFAS v 3.1) for 55 gages. Units are m3/sec. 1979-2022.glofas_23.zip - Glofas v3.1 file for the 23 permafrost-dominant gages in the study. Monthly, seasonal, and annual streamflow observations for minimum streamflow, mean streamflow, and maximum streamflow. Units are m3/sec. 1979-2021.glofas_1583.zip - Glofas v3.1 file for the 1583 randomly selected permafrost-dominant gages in the study. Monthly, seasonal, and annual streamflow observations for minimum streamflow, mean streamflow, and maximum streamflow. Units are m3/sec. 1979-2021.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS2011

ISS Operations Cost Reductions Through Automation of Real-Time Planning Tasks

In 2008 the Johnson Space Center s Mission Operations Directorate (MOD) management team challenged their organization to find ways to reduce the costs of International Space station (ISS) console operations in the Mission Control Center (MCC). Each MOD organization was asked to identify projects that would help them attain a goal of a 30% reduction in operating costs by 2012. The MOD Operations and Planning organization responded to this challenge by launching several software automation projects that would allow them to greatly improve ISS console operations and reduce staffing and operating costs. These projects to date have allowed the MOD Operations organization to remove one full time (7 x 24 x 365) ISS console position in 2010; with the plan of eliminating two full time ISS console support positions by 2012. This will account for an overall 10 EP reduction in staffing for the Operations and Planning organization. These automation projects focused on utilizing software to automate many administrative and often repetitive tasks involved with processing ISS planning and daily operations information. This information was exchanged between the ground flight control teams in Houston and around the globe, as well as with the ISS astronaut crew. These tasks ranged from managing mission plan changes from around the globe, to uploading and downloading information to and from the ISS crew, to even more complex tasks that required multiple decision points to process the data, track approvals and deliver it to the correct recipient across network and security boundaries. The software solutions leveraged several different technologies including customized web applications and implementation of industry standard web services architecture between several planning tools; as well as a engaging a previously research level technology (TRL 2-3) developed by Ames Research Center (ARC) that utilized an intelligent agent based system to manage and automate file traffic flow, archiving f data, and generating console logs. This technology called OCAMS (OCA (Orbital Communication System) Management System), is now considered TRL level 9 and is in daily use in the Mission Control Center in support of ISS operations. These solutions have not only allowed for improved efficiency on console; but since many of the previously manual data transfers are now automated, many of the human error prone steps have been removed, and the quality of the planning products has improved tremendously. This has also allowed our Planning Flight Controllers more time to focus on the abstract areas of the job, (like the complexities of planning a mission for 6 international crew members with a global planning team), instead of being burdened with the administrative tasks that took significant time each console shift to process. The resulting automation solutions have allowed the Operations and Planning organization to realize significant cost savings for the ISS program through 2020 and many of these solutions could be a viable

Hall, Timothy A.↗

DOE OSTI2024

Grid Resilience Plans: State Requirements, Utility Practices, and Utility Plan Template

As of June 2024, 14 states and one city require jurisdictional electric utilities to file resilience plans. Drawing on these requirements and filed plans, this report offers a standard template that states and utilities can consider to improve utility filings for grid resilience plans, either as part of a distribution system plan or as a separate filing. Key elements include a vulnerability assessment, description of proposed resilience programs, and projected costs and rate impacts. While many of these requirements and plans focus on extreme weather hazards, the template also can be used to address additional threats, including cyber and physical attacks and seismic events.

24 POWER TRANSMISSION AND DISTRIBUTION↗

NASA NTRS1980

Interactive display of molecular models using a microcomputer system

A simple, microcomputer-based, interactive graphics display system has been developed for the presentation of perspective views of wire frame molecular models. The display system is based on a TERAK 8510a graphics computer system with a display unit consisting of microprocessor, television display and keyboard subsystems. The operating system includes a screen editor, file manager, PASCAL and BASIC compilers and command options for linking and executing programs. The graphics program, written in USCD PASCAL, involves the centering of the coordinate system, the transformation of centered model coordinates into homogeneous coordinates, the construction of a viewing transformation matrix to operate on the coordinates, clipping invisible points, perspective transformation and scaling to screen coordinates; commands available include ZOOM, ROTATE, RESET, and CHANGEVIEW. Data file structure was chosen to minimize the amount of disk storage space. Despite the inherent slowness of the system, its low cost and flexibility suggests general applicability.

Egan, J. T.↗

DOE OSTI2016

PcapDB: Search Optimized Packet Capture, Version 0.1.0.0

PcapDB is a packet capture system designed to optimize the captured data for fast search in the typical (network incident response) use case. The technology involved in this software has been submitted via the IDEAS system and has been filed as a provisional patent. It includes the following primary components: capture: The capture component utilizes existing capture libraries to retrieve packets from network interfaces. Once retrieved the packets are passed to additional threads for sorting into flows and indexing. The sorted flows and indexes are passed to other threads so that they can be written to disk. These components are written in the C programming language. search: The search components provide a means to find relevant flows and the associated packets. A search query is parsed and represented as a search tree. Various search commands, written in C, are then used resolve this tree into a set of search results. The tree generation and search execution management components are written in python. interface: The PcapDB web interface is written in Python on the Django framework. It provides a series of pages, API's, and asynchronous tasks that allow the user to manage the capture system, perform searches, and retrieve results. Web page components are written in HTML,CSS and Javascript.

Ferrell, Paul↗