SEARCH · Engineering Papers

Results for “data formats”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

All sources NASA NTRS DOE OSTI

NASA NTRS2000

Issues Involved in the Development of an Open Standard for Data Link of Aviation Weather Information

This paper describes how an effective and efficient data link system for the dissemination of aviation weather information could be constructed. The system is built upon existing 'open standard' foundations drawn from current aviation and computer technologies. Issues of communications protocols and application data formats are discussed. The proposed aviation weather data link system is dependent of the actual link mechanism selected.

Grappel, R. D.↗

NASA NTRS2017

Online Visualization and Value Added Services of MERRA-2 Data at GES DISC

NASA climate reanalysis datasets from MERRA-2, distributed at the Goddard Earth Sciences Data and Information Services Center (GES DISC), have been used in broad research areas, such as climate variations, extreme weather, agriculture, renewable energy, and air quality, etc. The datasets contain numerous variables for atmosphere, land, and ocean, grouped into 95 products. The total archived volume is approximately 337 TB ( approximately 562K files) at the end of October 2017. Due to the large number of products and files, and large data volumes, it may be a challenge for a user to find and download the data of interest. The support team at GES DISC, working closely with the MERRA-2 science team, has created and is continuing to work on value added data services to best meet the needs of a broad user community. This presentation, using aerosol over Asia Monsoon as an example, provides an overview of the MERRA-2 data services at GES DISC, including: How to find the data? How many data access methods are provided? What are the best data access methods for me? How do download the subsetted (parameter, spatial, temporal) data and save in preferred spatial resolution and data format? How to visualize and explore the data online? In addition, we introduce a future online analytic tool designed for supporting application research, focusing on long-term hourly time-series data access and analysis.

Shen, Suhung↗

NASA NTRSDate not supplied

Design for Manufacturing of Structures with Automated Fiber Placement via Integration of Analysis and Process Planning

Under the Design for Manufacturing (DFM) task in the NASA HiCAM program, significant process has been made towards establishing a fully automated optimization process that spans the structural analysis, design, and manufacturing process planning for Automated Fiber Placement (AFP) structures. Previous efforts in this area established the data formats and mapping processes needed to exchange data between disciplines. The software tools used in the framework are HyperX (structural optimization), CAPP (process planning), and VCP (AFP path generation). The work has culminated with automation of data exchanges and an optimization process that drives the three software toward convergence. Within this framework, HyperX is used to generate mass-optimum composite designs which are iterated with VCP and CAPP to improve manufacturability of the designs. As the ply manufacturing process is adjusted by CAPP and VCP, HyperX continually re-assesses the structural integrity of the part and makes adjustments to the laminate as needed. For example, the position of fiber steering and tow overlaps/gaps can necessitate changes to the laminate design. Once the design converges, the end result is a stack of plies that meets both structural and manufacturing requirements. This paper presents the details of the framework and demonstrations the automated process on several parts with significant double-curvature.

HiCAM↗

DOE OSTI2023

A Ceph S3 Object Data Store for HEP

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2024

Efficient Parallelization of Irregular Applications on GPU Architectures

With the enlarging computation capacity of general Graphics Processing Units (GPUs), leveraging GPUs to accelerate parallel applications has become a critical topic in academia and industry. However, a wide range of irregular applications with the computation-/memory-intensive nature cannot easily achieve high GPU utilization. The challenges mainly involve the following aspects: first, data dependence leads to coarse-grained kernel and inefficient parallelism; second, heavy GPU memory usage may cause frequent memory evictions and extra overhead of I/O; third, specific computation patterns produce memory redundancies; last, workload balance and data reusability conjunctly benefit the overall performance, but there may exist a dynamic trade-off between them. Targeting these challenges, this dissertation proposes multiple optimizations to accelerate two real-world applications: many-body correlation functions to simulate nuclear physics in a large-scale scientific system; the other is the eALS-based matrix factorization recommendation system. To accelerate the calculations of many-body correlation functions, this dissertation presents three frameworks in GPU memory management and multi-GPU scheduling. Firstly, an optimized systematic GPU memory management framework, MemHC, utilizes a series of new memory reduction designs in GPU memory allocation, CPU/GPU communications, and GPU memory oversubscription. Secondly, an enhanced multi-GPU scheduling framework, MICCO, particularly by taking both data dimension (e.g., data reuse and data eviction) and computation dimension into account. MICCO designs a heuristic scheduling algorithm and a machine learning-based regression model to generate the optimal settings of a proposed new concept to manage the trade-off. Thirdly, a locality-aware multi-GPU scheduling framework. This scheduler leverages pipeline batch generation with a looking-ahead strategy by building local dependency graphs for memory transfer reduction and better data reuse, achieving up to 79.92% memory cost reduction and 1.67x speedup. To parallelize the eALS-based recommendation system, this dissertation proposes an efficient CPU/GPU heterogeneous recommendation system, HEALS. HEALS employs newly designed architecture-adaptive data formats to achieve load balance and good data locality on CPU and GPU. To mitigate the data dependence, HEALS presents a CPU/GPU collaboration model for both task parallelism and data parallelism with multiple kernel computation optimizations. In summary, this dissertation efficiently accelerates two typical irregular applications on GPUs by building four frameworks, including CPU/GPU collaboration, GPU memory management, and multi-GPU scheduling.

Wang, Qihan↗

NASA NTRS1993

An application protocol for CAD to CAD transfer of electronic information

The exchange of Computer Aided Design (CAD) information between dissimilar CAD systems is a problem. This is especially true for transferring electronics CAD information such as multi-chip module (MCM), hybrid microcircuit assembly (HMA), and printed circuit board (PCB) designs. Currently, there exists several neutral data formats for transferring electronics CAD information. These include IGES, EDIF, and DXF formats. All these formats have limitations for use in exchanging electronic data. In an attempt to overcome these limitations, the Navy's MicroCIM program implemented a project to transfer hybrid microcircuit design information between dissimilar CAD systems. The IGES (Initial Graphics Exchange Specification) format is used since it is well established within the CAD industry. The goal of the project is to have a complete transfer of microelectronic CAD information, using IGES, without any data loss. An Application Protocol (AP) is being developed to specify how hybrid microcircuit CAD information will be represented by IGES entity constructs. The AP defines which IGES data items are appropriate for describing HMA geometry, connectivity, and processing as well as HMA material characteristics.

Azu, Charles C., Jr.↗

NASA NTRS2020

Air Traffic Management TestBed Data Exchange Model

The Air Traffic Management (ATM) TestBed is a Platform as a Service that is being developed by the National Aeronautics and Space Administration (NASA) to help design, configure, integrate, run, and monitor air traffic simulations. The platform is designed to provide cloud services including back-end, big-data analytics tools, on-demand computing resource management, data storage, and communication middleware. The ATM TestBed reduces the time to test concepts and technologies, supports interactions among various methods such as human-in-the-loop and automation-in-the-loop simulations, and enables collaborative simulations by sharing technologies and tools in the ATM community. In order to allow easier access to simulation components, TestBed provides a messaging support layer for connectivity using a consistent set of input/output interfaces. In addition, a standard data format is introduced to facilitate communication between the components. The data exchange model, supported in the messaging support layer, standardizes the format of the information to be exchanged among the components. This document describes the messaging data model currently developed in TestBed and provides data dictionaries for references to component developers as well as simulation engineers.

air traffic simulation↗

DOE OSTI2019

Parallel Event Selection on HPC Systems

In their recent measurement of the neutrino oscillation parameters,NOvA uses a sample of approximately 25 million reconstructed spills to searchfor electron-neutrino appearance events. These events are stored in an n-tupleformat, in 250 thousand ROOT files. File sizes range from a few hundred KiB toa few MiB; the full dataset is approximately 1.4 TiB. These millions of eventsare reduced to a few tens of events by the application of strict event selectioncriteria, and then summarized by a handful of numbers each, which are used inthe extraction of the neutrino oscillation parameters.The NOvA event selection code is currently a serial C++ program that readsthese n-tuples. The current table data format and organization and the selection/reduction processing involved provides us with an opportunity to explorealternate approaches to represent the data and implement the processing. Werepresent our n-tuple data in HDF5 format that is optimized for the HPC environmentand which allows us to use the machine’s high-performance parallelI/O capabilities. We use MPI, numpy and h5py to implement our approach andcompare the performance with the existing approach. We study the performanceimplications of using thousands of small files of different sizes as comparedwith one large file using HPC resources. This work has been done as part of theSciDAC project, “HEP analytics on HPC” in collaboration with the ASCR teamsat ANL and LBNL.

Paterno, Marc↗

NASA NTRS2000

BOREAS Forest Cover Data Layers of the NSA in Raster Format

This data set was processed by BORIS staff from the original vector data of species, crown closure, cutting class, and site classification/subtype into raster files. The original polygon data were received from Linnet Graphics, the distributor of data for MNR. In the case of the species layer, the percentages of species composition were removed. This reduced the amount of information contained in the species layer of the gridded product, but it was necessary in order to make the gridded product easier to use. The original maps were produced from 1:15,840-scale aerial photography collected in 1988 over an area of the BOREAS NSA MSA. The data are stored in binary, image format files and they are available from Oak Ridge National Laboratory. The data files are available on a CD-ROM (see document number 20010000884).

Hall, Forrest G.↗

DOE OSTI2016

IDC Re-Engineering Phase 2 Data Model to IDC Schema Mapping.

This initial draft document contains formative data model content for select areas of Re-Engineering Phase 2 IDC System. The purpose of this document is to facilitate discussion among the stakeholders. It is not intended as a definitive proposal.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NASA NTRSDate not supplied

Data Democratization: Challenges and Opportunities

Democratizing Earth data is one of the challenges many organizations around the world face in order to maximize the use of their Earth data for research, applications, education, and societal benefits. For example, at the NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC), over 1600 global and regional datasets in several NASA Earth science focus areas, including atmospheric composition, water and energy cycles, and climate variability, are archived and distributed to the public. Giovanni, the Geospatial Interactive Online Visualization and Analysis Infrastructure, was developed by GES DISC to facilitate data access and exploration, especially for novice users of Earth science. With Giovanni, users can analyze and visualize over 2000 Earth science variables (e.g., precipitation, aerosol, surface wind) without downloading data, software, the expert understanding of data formats and structures, and coding skills, lowering the barrier to data analysis/comparison by preprocessing and accessing to the data. Results of data analysis and visualization can be accessed in several popular formats (e.g., NetCDF, CSV). As a result of Giovanni's efforts, more than 3000 referral papers have been published in various fields. In spite of this, Giovanni is still difficult to use for some users. For instance, if one searches for "precipitation," it will return over 150 related variables. The question is, which one to use? Furthermore, variables from different data providers (e.g., satellites and models) are named differently with different units, further confusing users, especially those outside the communities. Data democratization is complex and multifaceted. Challenges include service and data discovery, user experiences, visualization, data quality, trustworthiness, and more. In this presentation, we will examine Giovanni as an example of challenges and opportunities in developing data democratization services.

data democratization↗

NASA NTRS1979

SAMICS: Input data preparation

The Solar Array Manufacturing Industry Costing Standards (SAMICS) provide standard formats, data, assumptions, and procedures for estimating the price that a manufacturer would have to charge for the product of a specified manufacturing process sequence. A line-by-line explanation is given of those standard formats which describe the economically important characteristics of the manufacturing processes and the technological structure of the companies and the industry. This revision provides an updated presentation of Format A Process Description, consistent with the October 1978 version of that form. A checklist of items which should be entered on Format A as direct expenses is included.

Chamberlain, R. G.↗

DOE OSTI2020

Generation and representation of synthetic smart meter data

Advanced energy algorithms running at big-data scale will be necessary to identify, realize, and verify energy savings to meet government and utility goals of building energy efficiency. Any algorithm must be well characterized and validated before it is trusted to run at these scales. Smart meter data from real buildings will ultimately be required for the development, testing, and validation of these energy algorithms and processes. However, for initial development and testing, smart meter data are difficult to work with due to privacy restrictions, noise from unknown sources, data accessibility, and other concerns which can complicate algorithm development and validation. This paper describes a new methodology to generate synthetic smart meter data of electricity use in buildings using detailed building energy modeling, which aims to capture the variability and stochastics of real energy use in buildings. The methodology can create datasets tailored to represent specific scenarios with known truth and controllable amounts of synthetic noise. Knowledge of ground truth also allows the development and validation of enhanced processes which leverage building metadata, such as building type or size (floor area), in addition to smart meter data. The methodology described in this paper includes the key influencing factors of real-world building energy use including weather data, occupant-driven loads, building operation and maintenance practices, and special events. Data formats to support workflows leveraging both synthetic meter data and associated metadata are proposed and discussed. Finally, example use cases of the synthetic meter data are described to illustrate potential applications.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2024

A Comprehensive Northern Hemisphere Particle Microphysics Data Set From the Precipitation Imaging Package

Microphysical observations of precipitating particles are critical data sources for numerical weather prediction models and remote sensing retrieval algorithms. However, obtaining coherent data sets of particle microphysics is challenging as they are often unindexed, distributed across disparate institutions, and have not undergone a uniform quality control process. This work introduces a unified, comprehensive Northern Hemisphere particle microphysical data set from the National Aeronautics and Space Administration precipitation imaging package (PIP), accessible in a standardized data format and stored in a centralized, public repository. Data is collected from 10 measurement sites spanning 34° latitude (37°N–71°N) over 10 years (2014–2023), which comprise a set of 1,070,000 precipitating minutes. The provided data set includes measurements of a suite of microphysical attributes for both rain and snow, including distributions of particle size, vertical velocity, and effective density, along with higher-order products including an approximation of volume-weighted equivalent particle densities, liquid equivalent snowfall, and rainfall rate estimates. The data underwent a rigorous standardization and quality assurance process to filter out erroneous observations to produce a self-describing, scalable, and achievable data set. Case study analyses demonstrate the capabilities of the data set in identifying physical processes like precipitation phase-changes at high temporal resolution. Bulk precipitation characteristics from a multi-site intercomparison also highlight distinct microphysical properties unique to each location. This curated PIP data set is a robust database of high-quality particle microphysical observations for constraining future precipitation retrieval algorithms, and offers new insights toward better understanding regional and seasonal differences in bulk precipitation characteristics.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

MolViewSpec: a Mol* extension for describing and sharing molecular visualizations

Data visualization is a pivotal component of a structural biologist’s arsenal. The Mol* Viewer makes molecular visualizations available to broader audiences via most web browsers. While Mol* provides a wide range of functionality, it has a steep learning curve and is only available via a JavaScript interface. To enhance the accessibility and usability of web-based molecular visualization, we introduce MolViewSpec (molstar.org/mol-view-spec), a standardized approach for defining molecular visualizations that decouples the definition of complex molecular scenes from their rendering. Scene definition can include references to commonly used structural, volumetric, and annotation data formats together with a description of how the data should be visualized and paired with optional annotations specifying colors, labels, measurements, and custom 3D geometries. Developed as an open standard, this solution paves the way for broader interoperability and support across different programming languages and molecular viewers, enabling more streamlined, standardized, and reproducible visual molecular analyses. MolViewSpec is freely available as a Mol* extension and a standalone Python package.

Midlik, Adam [European Bioinformatics Institute (U↗

DOE OSTI2021

Creating a Tools Ecosystem for Cross-Discipline Environmental Data Reuse

Reusing data is difficult even within well-defined science communities and only gets worse when combining data from multiple communities and disciplines. Through the lens of current work on constructing an environmental epidemiological data set from multiple disciplinary sources, we demonstrate the need for a new tool ecosystem to support heterogeneous Big Data science. Extending existing community standards for schemas and/or data formats through human auditing and wrangling of the data is not feasible at scale. This work therefore suggests new approaches for the multi-disciplinary communities to build a shared tool ecosystem for big data. We discuss both the larger context of data wrangling of epidemiological data sets for novel artificial intelligence algorithms and the specific lessons from working with these multi-disciplinary data sets. Adopting a more model-driven, automatable approach promises not only better efficiency but also removes key sources of human-generated errors and promotes reuse and reproducibility of science data.

Logan, Jeremy↗

DOE OSTI2022

Spiner: Performance Portable Routines for Generic, Tabulated, Multi-Dimensional Data

We present Spiner, a new, performance-portable library for working with tabulated data. Spiner provides efficient routines for multi-dimensional interpolation and indexing on both CPUs and GPUs—as well as more exotic hardware—including interwoven interpolation and indexing access patterns, as needed for radiation transport. Importantly, Spiner defines a data format, based on HDF5, that couples the tabulated data to the information required to interpolate it, which Spiner can read and move to GPU.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2019

WPEC Subgroup-38 Final Report part II: Specifications for a new database structure

This is the revision of the specifications for the Generalised Nuclear Database Structure (GNDS), corresponding to GNDS version 1.9. GNDS is designed to serve as a replacement for the Evaluated Nuclear Data Format (ENDF). Although GNDS supersedes the ENDF6 format, GNDS should be viewed as a natural evolution of the ENDF-6 format, using modern computational tools to make nuclear data more accessible.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗