SEARCH · Engineering Papers

Results for “Distributed File Systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20

All sources NASA NTRS DOE OSTI

NASA NTRSDate not supplied

Simplifying Analysis of Hierarchical HDF5 and NetCDF4 Files with Xarray-Datatree

NASA’s Earth Observing System Data and Information System (EOSDIS) contains thousands of Earth science datasets from satellites, models, and field campaigns. EOSDIS data are stored in formats that are well supported by the Earth Science community. These formats include the Hierarchical Data Format (HDF), with derivative flavors such as HDF-5 and the Network Common Data Format (NetCDF-4). The HDF specification allows for a directory-like hierarchy within a single file, known as "groups". Observational data and associated metadata within a single file can be distributed amongst multiple internal groups, which can also be nested to multiple levels. Working with datasets that have a group hierarchical structure can be difficult because of the nested structure of groups. Widely used packages, such as xarray, have data models that do not accommodate the hierarchical structure within HDF files, requiring users to traverse the file and open different HDF groups as separate, unrelated objects. Xarray-datatree is a Python package developed to solve the difficulty of traversing HDFs with a hierarchical group structure by creating a tree-like hierarchical data structure in xarray. The tree-like structure allows each group to be accessed once a DataTree object is instantiated. The migration of xarray-datatree into the xarray core library will reduce barriers to accessing Earth science data by eliminating the need to understand and traverse the specific hierarchy of a grouped HDF file.

Eni Awowale↗

NASA NTRS2000

BOREAS TE-1 Soils Data Over The SSA Tower Sites in Raster Format

The BOREAS TE-1 team collected various data to characterize the soil-plant systems in the BOREAS SSA. This data set was gridded from vector layers of soil maps that were received from Dr. Darwin Anderson (TE-1), who did the original soil mapping in the field during 1994. The vector layers were gridded into raster files that cover approximately 1 square kilometer over each of the tower sites in the SSA. The data files are available on a CD-ROM (see document number 20010000884), or from the Oak Ridge National Laboratory (ORNL) Distributed Active Archive Center (DAAC).

Hall, Forrest G.↗

NASA NTRS2013

MISSE in the Materials and Processes Technical Information System (MAPTIS )

Materials International Space Station Experiment (MISSE) data is now being collected and distributed through the Materials and Processes Technical Information System (MAPTIS) at Marshall Space Flight Center in Huntsville, Alabama. MISSE data has been instrumental in many programs and continues to be an important source of data for the space community. To facilitate great access to the MISSE data the International Space Station (ISS) program office and MAPTIS are working to gather this data into a central location. The MISSE database contains information about materials, samples, and flights along with pictures, pdfs, excel files, word documents, and other files types. Major capabilities of the system are: access control, browsing, searching, reports, and record comparison. The search capabilities will search within any searchable files so even if the desired meta-data has not been associated data can still be retrieved. Other functionality will continue to be added to the MISSE database as the Athena Platform is expanded

Burns, DeWitt↗

DOE OSTI2020

Automatic log analysis with NLP for the CMS workflow handling

The central Monte-Carlo production of the CMS experiment utilizes the WLCG infrastructure and manages daily thousands of tasks, each up to thousands of jobs. The distributed computing system is bound to sustain a certain rate of failures of various types, which are currently handled by computing operators a posteriori. Within the context of computing operations, and operation intelligence, we propose a Machine Learning technique to learn from the operators with a view to reduce the operational workload and delays. This work is in continuation of CMS work on operation intelligence to try and reach accurate predictions with Machine Learning. We present an approach to consider the log files of the workflows as regular text to leverage modern techniques from Natural Language Processing (NLP). In general, log files contain a substantial amount of text that is not human language. Therefore, different log parsing approaches are studied in order to map the log files’ words to high dimensional vectors. These vectors are then exploited as feature space to train a model that predicts the action that the operator has to take. This approach has the advantage that the information of the log files is extracted automatically and the format of the logs can be arbitrary. In this work the performance of the log file analysis with NLP is presented and compared to previous approaches.

Layer, Lukas↗

DOE OSTI2024

Characterization of Soil Thermal and Electrical Properties along Multiple Hillslope Transects at Teller Road Site, Seward Peninsula, Alaska, 2017

This dataset has been acquired along five-119 m long transects located on the bottom part of the watershed hillslope at the NGEE Arctic Teller Road site at mile marker 27 (TL_MM27) on the Seward Peninsula, Alaska in July and September 2017. The Distributed Temperature Profiling (DTP) system dataset consist in vertically-resolved profile of soil temperature covering the top 0.8 m of soil with 8 cm interval. In addition to DPT data, electrical resistivity tomography (ERT) data, soil moisture, depth to rock or thaw layer thickness (no differentiation) and ground elevations data have been acquired along each of the transects. A UAV-based geotiff mosaic of the investigated site is also provided. The four data types provided with this dataset of 37 files (*.csv, *.tif, *.DATA): (1) soil temperature profiles, (2) ERT data, (3) the physical measurements of the thaw layer, and (4) an orthomosaic GeoTIFF of the transect study area.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS1995

Using dark current data to estimate AVIRIS noise covariance and improve spectral analyses

Starting in 1994, all AVIRIS data distributions include a new product useful for quantification and modeling of the noise in the reported radiance data. The 'postcal' file contains approximately 100 lines of dark current data collected at the end of each data acquisition run. In essence this is a regular spectral-image cube, with 614 samples, 100 lines and 224 channels, collected with a closed shutter. Since there is no incident radiance signal, the recorded DN measure only the DC signal level and the noise in the system. Similar dark current measurements, made at the end of each line are used, with a 100 line moving average, to remove the DC signal offset. Therefore, the pixel-by-pixel fluctuations about the mean of this dark current image provide an excellent model for the additive noise that is present in AVIRIS reported radiance data. The 61,400 dark current spectra can be used to calculate the noise levels in each channel and the noise covariance matrix. Both of these noise parameters should be used to improve spectral processing techniques. Some processing techniques, such as spectral curve fitting, will benefit from a robust estimate of the channel-dependent noise levels. Other techniques, such as automated unmixing and classification, will be improved by the stable and scene-independence noise covariance estimate. Future imaging spectrometry systems should have a similar ability to record dark current data, permitting this noise characterization and modeling.

Boardman, Joseph W.↗

NASA NTRS1978

User's manual: Subsonic/supersonic advanced panel pilot code

Sufficient instructions for running the subsonic/supersonic advanced panel pilot code were developed. This software was developed as a vehicle for numerical experimentation and it should not be construed to represent a finished production program. The pilot code is based on a higher order panel method using linearly varying source and quadratically varying doublet distributions for computing both linearized supersonic and subsonic flow over arbitrary wings and bodies. This user's manual contains complete input and output descriptions. A brief description of the method is given as well as practical instructions for proper configurations modeling. Computed results are also included to demonstrate some of the capabilities of the pilot code. The computer program is written in FORTRAN IV for the SCOPE 3.4.4 operations system of the Ames CDC 7600 computer. The program uses overlay structure and thirteen disk files, and it requires approximately 132000 (Octal) central memory words.

Moran, J.↗

DOE OSTI2023

FERC Order No. 2222 and Considerations for Distributed Wind

The Federal Energy Regulatory Commission (FERC) issued Order No. 2222 in October 2020. The rule directs Regional Transmission Organizations and Independent System Operators (ISOs) to amend their tariffs and participation models to accommodate heterogeneous distributed energy resource (DER) aggregations in the wholesale energy markets that they operate, including capacity, energy, and ancillary service markets. The Commission issued the rule to better capture the benefits provided by DERs deployed in the United States, whose use has been expanding rapidly. The Commission defines DERs as “any resource located on the distribution system, any subsystem thereof or behind a customer meter,” including but not limited to “electric storage resources, distributed generation, demand response, energy efficiency, thermal storage, and electric vehicles and their supply equipment.” The rule aims to increase DER participation in wholesale markets by allowing the creation of DER aggregations, in which multiple DERs that are too small to meet minimum capacity requirements for wholesale markets individually would be able to participate in markets as a single unit. As of June 20, 2023, all ISOs have filed initial compliance plans and a number have begun implementation. Compliance dates range from 2024 to 2029, with Midcontinent ISO having the latest date of compliance proposed for 2029. Southwest Power Pool still has an outstanding date, having no final order yet from FERC, but a target date of the third quarter of calendar year 2025. The rule, which is technology agnostic and requires ISOs to create participation plans that accommodate different DERs, provides an opportunity for distributed wind market expansion. In addition, distributed wind can bring benefits to heterogenous DER aggregations. These benefits include resource diversity (i.e., a complementary generation profile to other types of distributed generation), its small footprint and ability to be co-located with load, and its potential to provide frequency response, voltage support, and black start services, among other ancillary services. This report provides a status update on FERC Order No. 2222, the current state of ISO compliance, and information relevant to the distributed wind industry as DER aggregators and other stakeholders expand their participation to wholesale energy markets.

17 WIND ENERGY↗

DOE OSTI2024

A Framework for Addressing Hydropower Modeling Gaps in Electric Grid Planning and Operational Studies [Slides]

Realistic representation of hydropower in power system planning and operation studies is extremely important as it helps in avoiding under/over-estimation of the services the hydro power units can provide. A framework and tools have been developed to account for hydrological conditions and modify power system models accordingly. For example, HyDat – data and AI driven platform, that provides that data required for editing the power system model files Collaborative effort with V&R Energy. Also, development of POM based tools have been performed, a) Tool1: editing .raw files with realistic current and maximum hydrogeneration and redispatch of other units, and b) Tool2: Editing .dyr files to represent water head, current and maximum generation. Finally the framework, HASP framework combines HyDat and V&R energy tools to produce modified power system model files. Key Take-away points from contingency analysis studies include a) the number of critical contingencies increases as the water levels are reduced (70%>75%>80%), and b) both the extent and the quality of dynamic response of the Hydropower unit is different with varying water head levels.

13 HYDRO ENERGY↗

NASA NTRS1999

A Collaborative Analysis Tool for Thermal Protection Systems for Single Stage to Orbit Launch Vehicles

Presented is a design tool and process that connects several disciplines which are needed in the complex and integrated design of high performance reusable single stage to orbit (SSTO) vehicles. Every system is linked to every other system and in the case of SSTO vehicles with air breathing propulsion, which is currently being studied by the National Aeronautics and Space Administration (NASA); the thermal protection system (TPS) is linked directly to almost every major system. The propulsion system pushes the vehicle to velocities on the order of 15 times the speed of sound in the atmosphere before pulling up to go to orbit which results high temperatures on the external surfaces of the vehicle. Thermal protection systems to maintain the structural integrity of the vehicle must be able to mitigate the heat transfer to the structure and be lightweight. Herein lies the interdependency, in that as the vehicle's speed increases, the TPS requirements are increased. And as TPS masses increase the effect on the propulsion system and all other systems is compounded. To adequately determine insulation masses for a vehicle such as the one described above, the aeroheating loads must be calculated and the TPS thicknesses must be calculated for the entire vehicle. To accomplish this an ascent or reentry trajectory is obtained using the computer code Program to Optimize Simulated Trajectories (POST). The trajectory is then used to calculate the convective heat rates on several locations on the vehicles using the Miniature Version of the JA70 Aerodynamic Heating Computer Program (MINIVER). Once the heat rates are defined for each body point on the vehicle, then insulation thicknesses that are required to maintain the vehicle within structural limits are calculated using Systems Improved Numerical Differencing Analyzer (SINDA) models. If the TPS masses are too heavy for the performance of the vehicle the process may be repeated altering the trajectory or some other input to reduce the TPS mass. The problem described is an example of the need for collaborative design and analysis. Analysis tools are being developed to facilitate these collaborative efforts. RECIPE is a cross-platform application capable of hosting a number of engineers and designers across the Internet for distributed and collaborative engineering environments. Such integrated system design environments allow for collaborative team design analysis for performing individual or reduced team studies. The analysis tools mentioned earlier are commonly run on different platforms and are usually run by different people. To facilitate the larger number of potential runs that may need to be made, RECIPE connects the computer codes that calculate the trajectory data, heat rate data, and TPS masses so that the output from each tool is easily transferred to the model input files that need it. This methodology is being applied to solve launch vehicle thermal design problems to shorten the design cycle, and enable the project team to evaluate design options. Results will be presented indicating the effectiveness of this as a collaborative design tool.

Alexander, Reginald A.↗

NASA NTRS2018

Important Changes to User Access at the NASA CDDIS

The Crustal Dynamics Data Information System (CDDIS) supports data archiving and distribution activities for the space geodesy and geodynamics community. The main objectives of the system are to make space geodesy and geodynamics related data and derived products available in a central archive, to maintain information about the archival of these data, to disseminate these data and information in a timely manner to a global scientific research community, and to provide user based tools for the exploration and use of the archive. Since its inception, the user community has utilized anonymous ftp for accessing and downloading files from the CDDIS archive. Although this protocol allows users to easily automate file downloads, many organizations, data systems, and users have already migrated from ftp or are actively pursuing a move away from the protocol due to problems from a system and security standpoint. Furthermore, U.S. Government agencies have become increasingly concerned about this legacy protocol and ensuring data integrity for the user community have begun recently to disallow the use of the ftp protocol. The CDDIS, operated by NASA GSFC, must therefore address these concerns and provide alternative methods for access to its archive for continued easy and automated download of its contents. This poster will discuss the upcoming changes at CDDIS and provide examples on transitioning from anonymous ftp.

Noll, Carey E.↗

NASA NTRS1980

Preliminary design of a future integrated design system

IPAD is a system of computer programs and data supporting the aerospace-vehicle design process by providing a set of services to aid in the management of a design project, project technical work, and project support work. Its purpose is to integrate people, programs, and data into a unified aerospace-vehicle design system. All project-management and technical data, together with certain standard data, are stored in a data base. The IPAD functions allow project personnel to query the data base and to perform operations on the data. This permits the orderly sequencing of the task elements of a complex operation and provides common access to a single data base by various participating groups who otherwise would require many separate files. These capabilities will be provided on a single host computer or across multiple heterogeneous computers on a distributed progress basis.

Diggins, R. M.↗

DOE OSTI2020

Pufferscale: Rescaling HPC Data Services for High Energy Physics Applications

User-space HPC data services are emerging as an appealing alternative to traditional parallel file systems, because of their ability to be tailored to application needs while eliminating unnecessary overheads incurred by POSIX compliance. The High Energy Physics (HEP) community is progressively turning towards such services to enable high-throughput accesses under heavy concurrency to billions of event data produced by instruments and consumed by subsequent analysis workflows. Such services would benefit from the possibility to be rescaled up and down to adapt to changing workloads, as experimental campaigns progress, in order to optimize resource usage. This paper formalizes rescaling a distributed storage system as a multi objective optimization problem considering three criteria: load balance, data balance, and duration of the rescaling operation. We propose a heuristic for rapidly finding a good approximate solution, while allowing users to weight the criteria as needed. The heuristic is evaluated with Pufferscale, a new rescaling manager for microservice-based distributed storage systems. To validate our approach in a real-world ecosystem, we showcase the use of Pufferscale as a means to enable storage malleability in the HEPnOS storage system for HEP applications.

Distributed Storage System↗

NASA NTRS1998

Incorporating the APS Catalog of the POSS I and Image Archive in ADS

The primary purpose of this contract was to develop the software to both create and access an on-line database of images from digital scans of the Palomar Sky Survey. This required modifying our DBMS (called Star Base) to create an image database from the actual raw pixel data from the scans. The digitized images are processed into a set of coordinate-reference index and pixel files that are stored in run-length files, thus achieving an efficient lossless compression. For efficiency and ease of referencing, each digitized POSS I plate is then divided into 900 subplates. Our custom DBMS maps each query into the corresponding POSS plate(s) and subplate(s). All images from the appropriate subplates are retrieved from disk with byte-offsets taken from the index files. These are assembled on-the-fly into a GIF image file for browser display, and a FITS format image file for retrieval. The FITS images have a pixel size of 0.33 arcseconds. The FITS header contains astrometric and photometric information. This method keeps the disk requirements manageable while allowing for future improvements. When complete, the APS Image Database will contain over 130 Gb of data. A set of web pages query forms are available on-line, as well as an on-line tutorial and documentation. The database is distributed to the Internet by a high-speed SGI server and a high-bandwidth disk system. URL is http://aps.umn.edu/IDB/. The image database software is written in perl and C and has been compiled on SGI computers with MIX5.3. A copy of the written documentation is included and the software is on the accompanying exabyte tape.

Humphreys, Roberta M.↗

DOE OSTI2022

Outdoor Test Bed Performance of a Power Line Sensor Using a Real-Time Event Simulator

This report summarizes the design and application of an outdoor power line sensor testbed (OPLST) with a real-time simulator and power meter to compare a potential transformer (PT) and current transformer (CT) versus an advanced line-post sensor. The OPLST was created to validate advanced medium-voltage (20/34.5 kV) outdoor power line sensors used in electrical distribution systems. Electrical utilities have installed metering and relay-protection transformers like PTs and CTs for several decades. The PTs/CTs are iron core measurement transformers based on the electromagnetic induction principle and provide reliable data in normal grid operation. However, new outdoor power line sensors (OPLS) using other technologies like voltage divider, Rogowski coil and optical principles have become available and may have favorable performance and costs compared to PTs/CTs. Therefore, the importance of testing these technologies with the PT/CT, to compare the measured phase voltage/current at different power grid scenarios is crucial to understand the performance of these new OPLS. For this study a G&W Electric Model CVS-36-O power line sensor was chosen as the OPLS to test with voltage/current signals. An OPAL-RT Technologies Model OP4510 real-time simulator and Schweitzer Engineering Laboratories Model SEL-735 power meter were installed with the 20/34.5 kV OPLST to compare the measured transient events collected from an advanced OPLS and the PT/CT. This system is installed at the Distributed Energy Communications and Control (DECC) lab, Oak Ridge National Laboratory (ORNL). The simulator generated different power grid scenarios (electrical faults, capacitor bank operation, service restoration, etc.), and its analog-output signals were connected to the voltage/current amplifiers that feed the 20/34.5 kV aerial cable loop through the PT/CT devices. Additional PT/CT devices were also wired with the medium voltage aerial cable loop to measure the phase current/voltage signals and server as references. After each test, common format for transient data exchange (COMTRADE) files were collected from the SEL-735 power meter and used to compare the performance of the OPLS with the PT/CT. The behavior of analog signals, harmonic components, total harmonic distortion and crest factors were assessed and found favorable for the G&W sensor as compared to the reference PT/CT.

24 POWER TRANSMISSION AND DISTRIBUTION↗

NASA NTRS2004

Digital Archive Issues from the Perspective of an Earth Science Data Producer

Contents include the following: Introduction. A Producer Perspective on Earth Science Data. Data Producers as Members of a Scientific Community. Some Unique Characteristics of Scientific Data. Spatial and Temporal Sampling for Earth (or Space) Science Data. The Influence of the Data Production System Architecture. The Spatial and Temporal Structures Underlying Earth Science Data. Earth Science Data File (or Relation) Schemas. Data Producer Configuration Management Complexities. The Topology of Earth Science Data Inventories. Some Thoughts on the User Perspective. Science Data User Communities. Spatial and Temporal Structure Needs of Different Users. User Spatial Objects. Data Search Services. Inventory Search. Parameter (Keyword) Search. Metadata Searches. Documentation Search. Secondary Index Search. Print Technology and Hypertext. Inter-Data Collection Configuration Management Issues. An Archive View. Producer Data Ingest and Production. User Data Searching and Distribution. Subsetting and Supersetting. Semantic Requirements for Data Interchange. Tentative Conclusions. An Object Oriented View of Archive Information Evolution. Scientific Data Archival Issues. A Perspective on the Future of Digital Archives for Scientific Data. References Index for this paper.

Barkstrom, Bruce R.↗

NASA NTRS2005

Lessons Learned From Developing Three Generations of Remote Sensing Science Data Processing Systems

The Biospheric Information Systems Branch at NASA s Goddard Space Flight Center has developed three generations of Science Investigator-led Processing Systems for use with various remote sensing instruments. The first system is used for data from the MODIS instruments flown on NASA s Earth Observing Systems @OS) Terra and Aqua Spacecraft launched in 1999 and 2002 respectively. The second generation is for the Ozone Measuring Instrument flying on the EOS Aura spacecraft launched in 2004. We are now developing a third generation of the system for evaluation science data processing for the Ozone Mapping and Profiler Suite (OMPS) to be flown by the NPOESS Preparatory Project (NPP) in 2006. The initial system was based on large scale proprietary hardware, operating and database systems. The current OMI system and the OMPS system being developed are based on commodity hardware, the LINUX Operating System and on PostgreSQL, an Open Source RDBMS. The new system distributes its data archive across multiple server hosts and processes jobs on multiple processor boxes. We have created several instances of this system, including one for operational processing, one for testing and reprocessing and one for applications development and scientific analysis. Prior to receiving the first data from OMI we applied the system to reprocessing information from the Solar Backscatter Ultraviolet (SBUV) and Total Ozone Mapping Spectrometer (TOMS) instruments flown from 1978 until now. The system was able to process 25 years (108,000 orbits) of data and produce 800,000 files (400 GiB) of level 2 and level 3 products in less than a week. We will describe the lessons we have learned and tradeoffs between system design, hardware, operating systems, operational staffing, user support and operational procedures. During each generational phase, the system has become more generic and reusable. While the system is not currently shrink wrapped we believe it is to the point where it could be readily adopted, with substantial cost savings, for other similar tasks.

Tilmes, Curt↗

NASA NTRS1993

Washington Double Star Catalog Cross Index (1950 position sort)

A machine-readable version of the Washington Catalog of Visual Double Stars (WDS) was prepared in 1984 on the basis of a data file that was collected and maintained for more than a century by a succession of double-star observers. Although this catalog is being continually updated, a new copy for distribution is not expected to be available for a few years. The WDS contains DM numbers, but many of these are listed only in the notes, which makes it difficult to search for double-star information, except by position. Hence, a cross index that provides complete DM identifications is desirable, and it appears useful to add HD numbers for systems in that catalog. Aitken Double Star (ADS) numbers were retained from the WDS, but no attempt was made to correct these except for obvious errors.

Source record↗