Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “File System”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,261 records · Page 70
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

Reducing I/O variability using dynamic I/O path characterization in petascale storage systems

In petascale systems with a million CPU cores, scalable and consistent I/O performance is becoming increasingly difficult to sustain mainly because of I/O variability. Furthermore, the I/O variability is caused by concurrently running processes/jobs competing for I/O or a RAID rebuild when a disk drive fails. We present a mechanism that stripes across a selected subset of I/O nodes with the lightest workload at runtime to achieve the highest I/O bandwidth available in the system. In this paper, we propose a probing mechanism to enable application-level dynamic file striping to mitigate I/O variability. We also implement the proposed mechanism in the high-level I/O library that enables memory-to-file data layout transformation and allows transparent file partitioning using subfiling. Subfiling is a technique that partitions data into a set of files of smaller size and manages file access to them, making data to be treated as a single, normal file to users. Here, we demonstrate that our bandwidth probing mechanism can successfully identify temporally slower I/O nodes without noticeable runtime overhead. Experimental results on NERSC’s systems also show that our approach isolates I/O variability effectively on shared systems and improves overall collective I/O performance with less variation.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2001

Network Configuration Analysis for Formation Flying Satellites

The performance of two networks to support autonomous multi-spacecraft formation flying systems is presented. Both systems are comprised of a ten-satellite formation, with one of the satellites designated as the central or 'mother ship.' All data is routed through the mother ship to the terrestrial network. The first system uses a TCP/EP over ATM protocol architecture within the formation, and the second system uses the IEEE 802.11 protocol architecture within the formation. The simulations consist of file transfers using either the File Transfer Protocol (FTP) or the Simple Automatic File Exchange (SAFE) Protocol. The results compare the IP queuing delay, IP queue size and IP processing delay at the mother ship as well as end-to-end delay for both systems. In all cases, using IEEE 802.11 within the formation yields less delay. Also, the throughput exhibited by SAFE is better than FTP.

Knoblock, Eric J.↗
NASA NTRS2004

Visualization, Analysis and Subsetting Tools for EOS Aura Data Products in HDF-EOS5

Aura data products are among the first to use the new version 5 of the Hierarchical Data Format for the Earth Observing System, or HDF-EOS5. This presentation discusses the common HDF-EOS5 file layout that is adopted for most of the EOS Aura standard data products. Details of the various tools that can be used to access, visualize and subset these data will also be provided. Aura, the NASA Earth Observing System's atmospheric chemistry mission, was successfully launched July 15, 2004. The Aura spacecraft includes four instruments: the High Resolution Dynamics Limb Sounder (HIRDLS), the Microwave Limb Sounder (MLS), the Ozone Monitoring Instrument (OMI), and the Tropospheric Emission Spectrometer (TES). Data from the HIRDLS, MLS and OMI will be archived at the NASA Goddard Earth Sciences (GES) Distributed Active Archive Center (DAAC), while TES data will be archived at the NASA Langley Research Center DAAC. For more information see http://daac.gsfc.nasa.gov/.

Johnson, J.↗
NASA NTRS2006

Lessons Learned from Daily Uplink Operations during the Deep Impact Mission

The daily preparation of uplink products (commands and files) for Deep Impact was as problematic as the final encounter images were spectacular. The operations team was faced with many challenges during the six-month mission to comet Tempel One of the biggest difficulties was that the Deep Impact Flyby and Impactor vehicles necessitated a high volume of uplink products while also utilizing a new uplink file transfer capability. The Jet Propulsion Laboratory (JPL) Multi-Mission Ground Systems and Services (MGSS) Mission Planning and Sequence Team (MPST) had the responsibility of preparing the uplink products for use on the two spacecraft. These responsibilities included processing nearly 15,000 flight products, modeling the states of the spacecraft during all activities for subsystem review, and ensuring that the proper commands and files were uplinked to the spacecraft. To guarantee this transpired and the health and safety of the two spacecraft were not jeopardized several new ground scripts and procedures were developed while the Deep Impact Flyby and Impactor spacecraft were en route to their encounter with Tempel-1. These scripts underwent several adaptations throughout the entire mission up until three days before the separation of the Flyby and Impactor vehicles. The problems presented by Deep Impact's daily operations and the development of scripts and procedures to ease those challenges resulted in several valuable lessons learned. These lessons are now being integrated into the design of current and future MGSS missions at JPL.

uplink↗
NASA NTRS2003

TFaNS-Tone Fan Noise Design/Prediction System: Users' Manual TFaNS Version 1.5

TFaNS is the Tone Fan Noise Design/Prediction System developed by Pratt & Whitney under contract to NASA Glenn. The purpose of this system is to predict tone noise emanating from a fan stage including the effects of reflection and transmission by the rotor and stator and by the duct inlet and nozzle. The first version of this design system was developed under a previous NASA contract. Several improvements have been made to TFaNS. This users' manual shows how to run this new system. TFaNS consists of the codes that compute the acoustic properties (reflection and transmission coefficients) of the various elements and writes them to files, CUP3D Fan Noise Coupling Code that reads these files, solves the coupling problem, and outputs the desired noise predictions, and AWAKEN CFD/Measured Wake Postprocessor which reformats CFD wake predictions and/or measured wake data so they can be used by the system. This report provides information on code input and file structure essential for potential users of TFaNS.

Topol, David A.↗
DOE OSTI2023

Finite Element Analysis System Workflow Tools

A collection of MATLAB functions and class definitions called System Workflow Tools (SWFT) are available to semi-automate steps in the simulation process. Some of these steps are often simple and routine for smaller finite element models, but if done directly by an analyst can quickly become labor intensive, cumbersome, and error prone for larger, system level models. Some of SWFT’s capabilities demonstrated in this report includes writing Sierra input decks and processing Quantities of Interest (QOI) from results files. SWFT also writes scripts in order to utilize other software programs such as Cubit (separating system level CAD into subassemblies and components, creating nodesets and sidesets), DAKOTA (ensemble management), and ParaView (contour plots and animations). Detailed commands and workflows from mesh generation to report generation are provided as examples for analysts to utilize SWFT capabilities.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2000

Subsetting and Formatting Landsat-7 LOR ETM+ and Data Products

The Landsat-7 Processing System (LPS) processes Landsat-7 Enhanced Thematic Mapper (ETM+) instrument data into large, contiguous segments called "subintervals" and stores them in Level OR (LOR) data files. The LPS processed subinterval products must be subsetted and reformatted before the Level I processing systems can ingest them. The initial full subintervals produced by the LPS are stored mainly in HDF Earth Observing System (HDF-EOS) format which is an extension to the Hierarchical Data Format (HDF). The final LOR products are stored in native HDF format. Primarily the EOS Core System (ECS) and alternately the DAAC Emergency System (DES) subset the subinterval data for the operational Landsat-7 data processing systems. The HDF and HDF-EOS application programming interfaces (APIs) can be used for extensive data subsetting and data reorganization. A stand-alone subsetter tool has been developed which is based on some of the DES code. This tool makes use of the HDF and HDFEOS APIs to perform Landsat-7 LOR product subsetting and demonstrates how HDF and HDFEOS can be used for creating various configurations of full LOR products. How these APIs can be used to efficiently subset, format, and organize Landsat-7 LOR data as demonstrated by the subsetter tool and the DES is discussed.

Reid, Michael R.↗
DOE OSTI2020

Exploiting CRIC to streamline the configuration management of GlideinWMS factories for CMS support

GlideinWMS is a workload management and provisioning system that allows sharing computing resources distributed over independent sites. Based on the requests made by GlideinWMS frontends, a dynamically sized pool of resources is created by GlideinWMS pilot factories via pilot job submission to resource sites’ CEs. More than 400 CEs are currently serving more than ten virtual organizations through GlideinWMS, with CMS being the biggest user with 230 CEs. The complex configurations of the parameters defining resource requests, as submitted to those CEs, have been historically managed by manually editing a set of different XML files. New possibilities arise with CMS adopting the CRIC, an information system that collects, aggregates, stores, and exposes, among other things, computing resource data coming from various data providers. The paper will describe the challenges faced when CMS started to use CRIC to automatically generate the GlideinWMS factory configurations. The architecture of the prototype, and the ancillary tools developed to ease this transition, will be discussed. Finally, future plans and milestones will be outlined.

Dost, Jeffrey↗
NASA NTRS1989

Onboard shuttle on-line software requirements system: Prototype

The prototype discussed here was developed as proof of a concept for a system which could support high volumes of requirements documents with integrated text and graphics; the solution proposed here could be extended to other projects whose goal is to place paper documents in an electronic system for viewing and printing purposes. The technical problems (such as conversion of documentation between word processors, management of a variety of graphics file formats, and difficulties involved in scanning integrated text and graphics) would be very similar for other systems of this type. Indeed, technological advances in areas such as scanning hardware and software and display terminals insure that some of the problems encountered here will be solved in the near-term (less than five years). Examples of these solvable problems include automated input of integrated text and graphics, errors in the recognition process, and the loss of image information which results from the digitization process. The solution developed for the Online Software Requirements System is modular and allows hardware and software components to be upgraded or replaced as industry solutions mature. The extensive commercial software content allows the NASA customer to apply resources to solving the problem and maintaining documents.

Kolkhorst, Barbara↗
DOE OSTI2021

Optode Images, and Dissolved Oxygen Data from Sand and Sediment Flow-through Column Experiments associated with: "2-D Imaging of Dissolved Oxygen Concentration in Flow-Through Sediment Columns"

This data package is associated with the publication "2-D Imaging of Dissolved Oxygen Concentration in Flow-Through Sediment Columns" submitted to Frontiers in Water (Garayburu-Caruso et al., 2022). This work presents a novel application of planar optode technology to sediment columns that allows the study of well-constrained flow fields without the need to assume completely homogeneous flow. We introduce a flow-through column with the interior coated in an oxygen sensing film and a series of sampling ports as a unique tool capable of capturing changes in dissolved oxygen (DO) at high frequency (10 s) and spatial (< 1 mm) resolution across the length of a column. In addition, we provide custom scripts that allows image processing steps to be automated and consistently reproduced over a variety of experimental conditions. This dataset is comprised of four folders (1) 01_Riverbed_Sediment_Experiment, (2) 02_Sand_Experiment, (3) 01_Riverbed_Sediment_Raw_Images, and (4) 02_Sand_Experiment_Raw_Images. 01_Riverbed_Sediment_Experiment contains (1) a subfolder with csv files that contain statistical parameters from the linear regressions applied to each optode image, (2) a subfolder with csv files from DO data measured by in-line sensors, (3) a subfolder with csv files containing column and injection specific input parameters for the image processing script, and (4) a subfolder with scripts used to process the images and create associated figures. 02_Sand_Experiment contains (1) a subfolder with csv files containing column and injection-specific input parameters for the image processing script, and (2) a subfolder with scripts used to process the images and create associated figures. 01_Riverbed_Sediment_Raw_Images (spited in 3 parts due to size) contains a subfolder with raw red, and green images from the optode system during the MilliQ Injection, Vanillin Injection or Vanillin + Sampling Injection respectively. 02_Sand_Experiment_Raw_Images contains a subfolder with raw red, and green images from the optode system. Outside of the main folders there is a csv containing file-level metadata and a csv data dictionary defining column headers for all csv files contained in the data package.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2007

Automatic Command Sequence Generation

Automatic Sequence Generator (Autogen) Version 3.0 software automatically generates command sequences for the Mars Reconnaissance Orbiter (MRO) and several other JPL spacecraft operated by the multi-mission support team. Autogen uses standard JPL sequencing tools like APGEN, ASP, SEQGEN, and the DOM database to automate the generation of uplink command products, Spacecraft Command Message Format (SCMF) files, and the corresponding ground command products, DSN Keywords Files (DKF). Autogen supports all the major multi-mission mission phases including the cruise, aerobraking, mapping/science, and relay mission phases. Autogen is a Perl script, which functions within the mission operations UNIX environment. It consists of two parts: a set of model files and the autogen Perl script. Autogen encodes the behaviors of the system into a model and encodes algorithms for context sensitive customizations of the modeled behaviors. The model includes knowledge of different mission phases and how the resultant command products must differ for these phases. The executable software portion of Autogen, automates the setup and use of APGEN for constructing a spacecraft activity sequence file (SASF). The setup includes file retrieval through the DOM (Distributed Object Manager), an object database used to store project files. This step retrieves all the needed input files for generating the command products. Depending on the mission phase, Autogen also uses the ASP (Automated Sequence Processor) and SEQGEN to generate the command product sent to the spacecraft. Autogen also provides the means for customizing sequences through the use of configuration files. By automating the majority of the sequencing generation process, Autogen eliminates many sequence generation errors commonly introduced by manually constructing spacecraft command sequences. Through the layering of commands into the sequence by a series of scheduling algorithms, users are able to rapidly and reliably construct the desired uplink command products. With the aid of Autogen, sequences may be produced in a matter of hours instead of weeks, with a significant reduction in the number of people on the sequence team. As a result, the uplink product generation process is significantly streamlined and mission risk is significantly reduced. Autogen is used for operations of MRO, Mars Global Surveyor (MGS), Mars Exploration Rover (MER), Mars Odyssey, and will be used for operations of Phoenix. Autogen Version 3.0 is the operational version of Autogen including the MRO adaptation for the cruise mission phase, and was also used for development of the aerobraking and mapping mission phases for MRO.

Fisher, Forest↗
DOE OSTI2021

Aerial Measuring System - Analysis of the Releasable Data Set

This is a collection of spectral data files obtained from AMS flights over areas in and around Las Vegas and the Nevada National Security Site. These are being made available for anyone to analyze and compare results with those in the accompanying report. The data files are attached to the PDF as Excel CSV format files.

99 GENERAL AND MISCELLANEOUS↗
NASA NTRS1982

Interactive digital signal processor

The Interactive Digital Signal Processor (IDSP) is examined. It consists of a set of time series analysis Operators each of which operates on an input file to produce an output file. The operators can be executed in any order that makes sense and recursively, if desired. The operators are the various algorithms used in digital time series analysis work. User written operators can be easily interfaced to the sysatem. The system can be operated both interactively and in batch mode. In IDSP a file can consist of up to n (currently n=8) simultaneous time series. IDSP currently includes over thirty standard operators that range from Fourier transform operations, design and application of digital filters, eigenvalue analysis, to operators that provide graphical output, allow batch operation, editing and display information.

Mish, W. H.↗
DOE OSTI2017

Performance Analysis of Large Scale HPC Workflows for Earth System Models

Climate assessment reports, such as the Intergovernmental Panel on Climate Change’s 5th assesment report [IPCC AR5; IPCC et al., 2014] or the United States National Climate Assessment [U.S. Global Change Research Program, 2014], increasingly rely on many-ensemble climate projections from fully-coupled Earth system models (ESM) to asses possible future climate states. ESM participate in a large set of Coupled Model Intercomparson Projects (CMIP) to better understand their models and the climate system, so that they may provide credible climate projections. However, fully participating in these intercomparison exercises represents an extremely large computational personnel burden for modeling centers [Eyring et al., 2016, Williams et al., 2016a]. However, there has been little attention within the climate modeling community paid to the in situ performance of ESMs up until now. Recently a computational performance model intercomparison project (CPMIP) for CMIP6 has begun [Balaji et al., 2017], but there has not yet been an effort to gather, process, and analyze the historical performance data that has been saved within archived log files from previous modeling efforts, nor the development of performance analysis system capable of providing detailed statistical analyses or visualizations of this performance data. Therefore, we’ve begun a joint effort to analyze the historical and current performance data on both GFDL and OLCF modeling systems. Here, we have developed a preliminary performance analysis system to captue, analyize, and visualize end-to-end in situ performance data and applied it to NOAA/GFDL’s post-processing system. Through the course of the project a number of performance improvements have been made, with the steps necessary to handle the expected volume of data (Big Data) likely to come out of the CMIP6 exercises (TB to PB) provided. We have also begun developing a statistical workload model that will allow modeling centers to determined their expected scientific throughput in the context of expected resource (computational and personnel) availability and accounting for typical workflow disruptions. This will allow them to develop a CMIP participation strategy and maximize their level of participation in the exercises. We expect this infrastructure, and the data it produces, to ultimately allow us to identify performance issues and workflow errors that exist across Earth system modeling centers, and possibly HPC modeling centers in general. Center-independent solutions to workflow or performance issues would help alleviate the computational burden for CMIP6+ modeling projects and should realize both labor and cost savings benefits throughout the CMIP community.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1988

The Environment for Application Software Integration and Execution (EASIE) version 1.0. Volume 4: System installation and maintenance guide

The Environment for Application Software Integration and Execution (EASIE) provides both a methodology and a set of software utility programs to ease the task of coordinating engineering design and analysis codes. This document provides necessary information for installing the EASIE software on a host computer system. The target host is a DEX VAX running VMS version 4; host dependencies are noted when appropriate. Relevant directories and individual files are identified, and compile/load/execute sequences are specified. In the case of the data management utilities, database management system (DBMS) specific features are described in an effort to assist the maintenance programmer in converting to a new DBMS. The document also describes a sample EASIE program directory structure to guide the program implementer in establishing his/her application dependent environment.

Randall, Donald P.↗
NASA NTRS2005

Lessons Learned From Developing Three Generations of Remote Sensing Science Data Processing Systems

The Biospheric Information Systems Branch at NASA s Goddard Space Flight Center has developed three generations of Science Investigator-led Processing Systems for use with various remote sensing instruments. The first system is used for data from the MODIS instruments flown on NASA s Earth Observing Systems @OS) Terra and Aqua Spacecraft launched in 1999 and 2002 respectively. The second generation is for the Ozone Measuring Instrument flying on the EOS Aura spacecraft launched in 2004. We are now developing a third generation of the system for evaluation science data processing for the Ozone Mapping and Profiler Suite (OMPS) to be flown by the NPOESS Preparatory Project (NPP) in 2006. The initial system was based on large scale proprietary hardware, operating and database systems. The current OMI system and the OMPS system being developed are based on commodity hardware, the LINUX Operating System and on PostgreSQL, an Open Source RDBMS. The new system distributes its data archive across multiple server hosts and processes jobs on multiple processor boxes. We have created several instances of this system, including one for operational processing, one for testing and reprocessing and one for applications development and scientific analysis. Prior to receiving the first data from OMI we applied the system to reprocessing information from the Solar Backscatter Ultraviolet (SBUV) and Total Ozone Mapping Spectrometer (TOMS) instruments flown from 1978 until now. The system was able to process 25 years (108,000 orbits) of data and produce 800,000 files (400 GiB) of level 2 and level 3 products in less than a week. We will describe the lessons we have learned and tradeoffs between system design, hardware, operating systems, operational staffing, user support and operational procedures. During each generational phase, the system has become more generic and reusable. While the system is not currently shrink wrapped we believe it is to the point where it could be readily adopted, with substantial cost savings, for other similar tasks.

Tilmes, Curt↗
DOE OSTI2022

Constraints on Future Analysis Metadata Systems in High Energy Physics

In high energy physics (HEP), analysis metadata comes in many forms—from theoretical cross-sections, to calibration corrections, to details about file processing. Correctly applying metadata is a crucial and often time-consuming step in an analysis, but designing analysis metadata systems has historically received little direct attention. Among other considerations, an ideal metadata tool should be easy to use by new analysers, should scale to large data volumes and diverse processing paradigms, and should enable future analysis reinterpretation. This document, which is the product of community discussions organised by the HEP Software Foundation, categorises types of metadata by scope and format and gives examples of current metadata solutions. Important design considerations for metadata systems, including sociological factors, analysis preservation efforts, and technical factors, are discussed. A list of best practices and technical requirements for future analysis metadata systems is presented. These best practices could guide the development of a future cross-experimental effort for analysis metadata tools.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Dataset and scripts for manuscript "Using Neural Network Ensembles to Separate Ocean Biogeochemical and Physical Drivers of Phytoplankton Biogeography in Earth System Models"

Please note: The title of this version contains an updated title for the manuscript compared to the previous version of this dataset. This is only due to title updates during the peer review process for the manuscript. The zip file contains the scripts, functions, and source files for the manuscript titled "Using Neural Network Ensembles to Separate Ocean Biogeochemical and Physical Drivers of Phytoplankton Biogeography in Earth System Models." The manuscript has been submitted for peer review. Please consult the README file for information on the specifications of the files. These files may occasionally be updated to add annotations to the scripts to make them more user friendly and to correct any errors.

54 ENVIRONMENTAL SCIENCES↗