Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data formats”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 505 records · Page 28

Incorporating Ameriflux Data into LVT

This paper describes a new generic data reader that was developed in Fortran to handle the Ameriflux data for the LIS Verification Toolkit (LVT). Researchers at the Hydrological Sciences Branch of NASA Goddard Space Flight Center have created a high resolution land surface modeling and data assimilation system known as the Land Information System (LIS), which provides an infrastructure to integrate state-of-the-art land surface models, data assimilation algorithms, observations of land surface from satellite and remotely sensed platforms to provide estimates of land surface conditions such as soil moisture, evaporation, snowpack and runoff. These model predictions are typically evaluated by comparing them with data from observational networks. The observational data; however, are usually available in disparate data formats and require significant effort to process them into a structure amenable for use with the model data. The motivation to develop a uniform approach for land surface verification as a way to alleviate these processing efforts has led to the development of LVT which is designed to enable the rapid evaluation of land surface modeling and analysis products from LIS. LVT focuses on the use of observational datasets in their native format. As the formats of these datasets vary widely, a major part of LVT is creating programs to read and process the native datasets. The primary goal of this project is to enhance LVT capabilities by incorporating observational datasets from Ameriflux

Georgiev, Teodor↗

The future of legume genetic data resources: Challenges, opportunities, and priorities

Abstract Legumes, comprising one of the largest, most diverse, and most economically important plant families, are the subject of vibrant research and development worldwide. Continued improvement of legume crops will benefit from the recent proliferation of genetic (including genomic) resources; but the diversity, scale, and complexity of these resources presents challenges to those managing and using them. A workshop held in March of 2019 addressed questions of data resources and priorities for the legumes. The workshop identified various needs and recommendations: (a) Develop strategies to effectively store, integrate, and relate genetic resources collected in different projects. (b) Leverage information collected across many legume species by standardizing data formats and ontologies, improving the state of metadata about datasets, and increasing use of the FAIR data principles. (c) Advocate for the critical role that curators exercise in integrating complex datasets into databases and adding high value metadata that enable downstream analytics and facilitate practical applications. (d) Implement standardized software and database development practices to best leverage limited developer time and expertise gained from the various legume (and other) species. (e) Develop tools and databases that can manage genetic information for the world's plant genetic resources, enabling efficient incorporation of important traits into breeding programs. (f) Centralize information on databases, tools, and training materials and establish funding streams to support training and outreach.

Bauchet, Guillaume J.↗

Open access to high-level data and analysis tools in the CMS experiment at the LHC

The CMS experiment, in recognition of its commitment to data preservation and open access as well as to education and outreach, has made its first public release of high-level data under the CC0 waiver: up to half of the proton-proton collision data (by volume) at 7 TeV from 2010 in CMS Analysis Object Data format. CMS has prepared, in collaboration with CERN and the other LHC experiments, an open-data web portal based on Invenio. The portal provides access to CMS public data as well as to analysis tools and documentation for the public. The tools include an event display and histogram application that run in the browser. In addition a virtual machine containing a CMS software environment along with XRootD access to the data is available. Within the virtual machine the public can analyse CMS data, example code is provided. As a result, we describe the accompanying tools and documentation and discuss the first experiences of data use.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Solar PV, Wind Generation, and Load Forecasting Dataset for ERCOT 2018: Performance-Based Energy Resource Feedback, Optimization, and Risk Management (P.E.R.F.O.R.M.)

This report describes the Advanced Research Projects Agency-Energy Performance-Based Energy Resource Feedback, Optimization, and Risk Management (PERFORM) Electric Reliability Council of Texas (ERCOT) dataset consisting of load, solar, and wind deterministic and probabilistic forecasts at three timescales. This dataset consists of 1 year of time-coincident load, wind, and solar actuals and probabilistic forecasts for a region similar to ERCOT. All the data are stored in Hierarchical Data Format 5 (HDF5) files and have been uploaded to an Amazon Web Services repository. The ERCOT data set has 2 years (2017, 2018) of actuals and 1 year (2018) of probabilistic forecasts. These data are provided at various spatial (i.e., site-level, zone-level, and system-level) and temporal scales (i.e., day-ahead, intraday, and intra-hour). Specifically, data are provided for 125 existing wind sites, 22 existing solar sites, 139 proposed wind sites, and 204 proposed solar sites.

14 SOLAR ENERGY↗

Merged Observatory Data Files (MODFs): an integrated observational data product supporting process-oriented investigations and diagnostics

A large and ever-growing body of geophysical information is measured in campaigns and at specialized observatories as a part of scientific expeditions and experiments. These collections of observed data include many essential climate variables (as defined by the Global Climate Observing System) but are often distinguished by a wide range of additional non-routine measurements that are designed to not only document the state of the environment but also the drivers that contribute to that state. These field data are used not only to further understand environmental processes through observation-based studies but also to provide baseline data to test model performance and to codify understanding to improve predictive capabilities. To address the considerable barriers and difficulty in utilizing these diverse and complex data for observation–model research, the Merged Observatory Data File (MODF) concept has been developed. A MODF combines measurements from multiple instruments into a single file that complies with well-established data format and metadata practices and has been designed to parallel the development of corresponding Merged Model Data Files (MMDFs). Using the MODF and MMDF protocols will facilitate the evolution of model intercomparison projects into model intercomparison and improvement projects by putting observation and model data “on the same page” in a timely manner. The MODF concept was developed especially for weather forecast model studies in the Arctic. The surprisingly complex process of implementing MODFs in that context refined the concept itself. Thus, this article explains the concept of MODFs by providing details on the issues that were revealed and resolved during that first specific implementation. Detailed instructions are provided on how to make MODFs, and this article can be considered a MODF creation manual.

54 ENVIRONMENTAL SCIENCES↗

New Earth Science Data and Access Methods

NASA's Earth Science Enterprise, working with its domestic and international partners, provides scientific data and analysis to improve life here on Earth. NASA provides science data products that cover a wide range of physical, geophysical, biochemical and other parameters, as well as services for interdisciplinary Earth science studies. Management and distribution of these products is administered through the Earth Observing System Data and Information System (EOSDIS) Distributed Active Archive Centers (DAACs), which all hold data within a different Earth science discipline. This paper will highlight selected EOS datasets and will focus on how these observations contribute to the improvement of essential services such as weather forecasting, climate prediction, air quality, and agricultural efficiency. Emphasis will be placed on new data products derived from instruments on board Terra, Aqua and ICESat as well as new regional data products and field campaigns. A variety of data tools and services are available to the user community. This paper will introduce primary and specialized DAAC-specific methods for finding, ordering and using these data products. Special sections will focus on orienting users unfamiliar with DAAC resources, HDF-EOS formatted data and the use of desktop research and application tools.

Moses, John F.↗

Revamping Spacecraft Operational Intelligence

The EPOXI flight mission has been testing a new commercial system, Splunk, which employs data mining techniques to organize and present spacecraft telemetry data in a high-level manner. By abstracting away data-source specific details, Splunk unifies arbitrary data formats into one uniform system. This not only reduces the time and effort for retrieving relevant data, but it also increases operational visibility by allowing a spacecraft team to correlate data across many different sources. Splunk's scalable architecture coupled with its graphing modules also provide a solid toolset for generating data visualizations and building real-time applications such as browser-based telemetry displays.

Hwang, Victor↗

Investigating Access Performance of Long Time Series with Restructured Big Model Data

Data sets generated by models are substantially increasing in volume, due to increases in spatial and temporal resolution, and the number of output variables. Many users wish to download subsetted data in preferred data formats and structures, as it is getting increasingly difficult to handle the original full-size data files. For example, application research users such as those involved with wind or solar energy, or extreme weather events are likely only interested in daily or hourly model data at a single point (or for a small area) for a long time period, and prefer to have the data downloaded in a single file. With native model file structures, such as hourly data from NASA Modern-Era Retrospective analysis for Research and Applications Version-2 (MERRA-2), it may take over 10 hours for the extraction of parameters-of-interest at a single point for 30 years. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is exploring methods to address this particular user need. One approach is to create value-added data by reconstructing the data files. Taking MERRA-2 data as an example, we have tested converting hourly data from one-day-per-file into different data cubes, such as one-month, or one-year. Performance is compared for reading local data files and accessing data through interoperable services, such as OPeNDAP. Results show that, compared to the original file structure, the new data cubes offer much better performance for accessing long time series. We have noticed that performance is associated with the cube size and structure, the compression method, and how the data are accessed. An optimized data cube structure will not only improve data access, but also may enable better online analysis services

reanalysis↗

Providing Long-term Solar and Meteorological Resource Information from NASA Research through the GIS-Enabled POWER Web Services Portal

Solar and other renewable energy systems are optimized using geophysical parameters describing information about the solar resource and meteorological environments and how those environments may be changing in time. The Prediction of Worldwide Renewable Energy Resource (POWER) team at the NASA LaRC Research Center deployed its first version of the revitalized POWER geophysical parameter website that employs Esri Geographic Information System (GIS) tools. This web application provides access to both time series and climatological data sets spanning from a few days behind real-time back to the early 1980’s with a spatial resolution of 0.5 x 0.5 degree, thus extending over 30 years. The GIS tools enable, generate and store climatological averages using spatial queries and calculations in a spatial database resulting in greater accessibility to government agencies, industry and individuals. There is an API that provides data through a URL coding and can also invoked inside user software packages. Output data formats now include ASCII, CSV, geoTiff, JSON and .netCDF. Since May 2018, over 10.4 TB of data have been delivered to fulfill over 110 million data requests from 240,000 unique user IPs (https://power.larc.nasa.gov). This presentation provides an overview of this project and the current version of the new POWER web capabilities through to the end usage. Surface solar parameters are adapted from both the NASA/GEWEX (Global and Energy Water Cycle Exchange) SRB (Surface Radiation Budget) project and the CERES (Clouds and Earth’s Radiant Energy System) data products. The meteorological parameters are adapted from NASA’s MERRA-2 (Modern Era Retrospective-analysis for Research and Applications). We review uncertainty of various basic parameters using surface measurements. After the introduction, we specifically discuss various clusters of parameters for equator pointing tilted surfaces that provide for the estimation of optimal tilt angle by month and year. The discussion will emphasize a few examples of how solar and building engineers are using the data products. We will then preview the capabilities of Version 2 featuring examples of hourly solar and meteorological data products, customized user reports and expanded web mapping services from CERES data products. The web services provide a unique, expandable resource for renewable energy systems engineers to design, evaluate and optimize to environments worldwide.

Solar Resource↗

Systematic feature design for cycle life prediction of lithium-ion batteries during formation

Optimization of the formation step in lithium-ion battery manufacturing is challenging due to limited physical understanding of solid-electrolyte interphase formation and the long testing time (∼100 days) for cells to reach the end of life. We propose a systematic feature-design framework that requires minimal domain knowledge for accurate cycle life prediction during formation. By only using two simple Q (V) features designed from our framework, extracted from formation data without any additional diagnostic cycles, we achieved an average of 9.87% error for cycle life prediction. Here, the physics-based investigation guided by the two designed features shows that the voltage ranges identified by our framework capture the effects of formation temperature and microscopic-particle resistance heterogeneity. By designing highly predictive, robust, and interpretable features, our approach can accelerate industrial battery formation research, leveraging the interplay between data-driven feature design and mechanistic understanding.

25 ENERGY STORAGE↗

OASIS: Open-Source AI Software Infrastructure for Science-SBIR Phase I

OASIS: Open source AI Software Infrastructure for Science is developed for researchers in the scientific domain. OASIS provides a data API to ingest and serve scientific data formats and annotations within AI workflows. It delivers a unique integration of features such as coupling of data to AI models, scalable training, cloud deployment into a cohesive web and command-line interface, and state-of-the-art techniques to debug and enhance AI models.

Chaudhary, Aashish↗

Spacelab data analysis and interactive control study

The study consisted of two main tasks, a series of interviews of Spacelab users and a survey of data processing and display equipment. Findings from the user interviews on questions of interactive control, downlink data formats, and Spacelab computer software development are presented. Equipment for quick look processing and display of scientific data in the Spacelab Payload Operations Control Center (POCC) was surveyed. Results of this survey effort are discussed in detail, along with recommendations for NASA development of several specific display systems which meet common requirements of many Spacelab experiments.

Tarbell, T. D.↗

Medium-range objective predictions of thunderstorms on the McIDAS/CSIS interactive computer system

Until recently, all operational meteorological data has been made available to forecasters in a variety of different forms. Predictions based upon these different data formats have been complicated by the inability of forecasters to easily assimilate, in real-time, all data to provide an optimum decision regarding future weather occurrences. By March 1980, a joint NASA/NOAA effort had been initiated to develop the Centralized Storm Information System (CSIS). The primary objectives of this joint project are related to an improvement of the overall severe storm forecast and warning procedure and to a demonstration of the operational utility of techniques developed within the applied research community. CSIS is to utilize the Man Computer Interactive Data Access System (McIDAS). The present investigation is concerned with one of the first attempts to employ the CSIS system for the evaluation of a new research technique involving the prediction of thunderstorms over a forecast period of 12-48 hours.

Wilson, G. S.↗

Documentation for the machine-readable version of the Revised S210 Catalog of Far-Ultraviolet Objects (Page, Carruthers and Heckathorn 1982)

A detailed description of the machine-readable revised catalog as it is currently being distributed from the Astronomical Data Center is given. This catalog of star images was compiled from imagery obtained by the Naval Research Laboratory (NRL) Far-Ultraviolet Camera/Spectrograph (Experiments S201) operated from 21 to 23 April 1972 on the lunar surface during the Apollo 16 mission. The documentation includes a detailed data format description, a table of indigenous characteristics of the magnetic tape file, and a sample listing of data records exactly as they are presented in the machine-readable version.

Warren, W. H., Jr.↗

Preservation and Enhancement of the Spacewatch Data Archives

In March of 1998, the asteroid 1997 XF11 was announced to be potentially hazardous after being tracked over 90 days. A potential two year wait for confirming observations was shortened to under 24 hours because of the existence of archived photographic prediscovery images. Spacewatch was a pioneer in using CCD scanning and possesses a valuable digital archive of its scans. Unfortunately these data are aging on magnetic tape and will soon be lost. Since 1990, the Spacewatch project gathered some 1.5 Terabytes of scan data covering roughly 75,000 degrees of sky to a limiting magnitude of V = 21.5. The data have not yet been mined for all of their asteroids for scientific studies and orbit determination. Spacewatch's real-time motion detection program MODP was constrained by the computers of the era to use simplified image processing algorithms at a reduced efficiency. Jedicke and Herron estimated MODP's efficiency at finding asteroids to be approximately 60 percent to V=18 and improving somewhat thereafter. This lead to a substantial bias correction in their analyses. Larsen has developed a MODP replacement capable in excess of 90 percent efficiency in the same range and able to push a magnitude fainter in completeness. We propose a program of post-processing and re-archiving Spacewatch data. Our scans would be transferred from tape to CD-ROMs and converted to FITS images -- establishing a consistent data format and media for both past and future Spacewatch observations. Larsen's MODP replacement would mine these data for previously undetected motions, which would be made available to the Minor Planet Center and our ongoing asteroid population studies. A searchable observation record would be made generally available for prediscovery work. We estimate the net asteroid yield of this proposal is equivalent to three full years of Spacewatch operations.

Larsen, Jeffrey A.↗

ISTP CDF Skeleton Editor

Basic Common Data Format (CDF) tools (e.g., cdfedit) provide no specific support for creating International Solar-Terrestrial Physics/Space Physics Data Facility (ISTP/SPDF) standard files. While it is possible for someone who is familiar with the ISTP/SPDF metadata guidelines to create compliant files using just the basic tools, the process is error-prone and unreasonable for someone without ISTP/SPDF expertise. The key problem is the lack of a tool with specific support for creating files that comply with the ISTP/SPDF guidelines. There are basic CDF tools such as cdfedit and skeletoncdf for creating CDF files, but these have no specific support for creating ISTP/ SPDF compliant files. The SPDF ISTP CDF skeleton editor is a cross-platform, Java-based GUI editor program that allows someone with only a basic understanding of the ISTP/SPDF guidelines to easily create compliant files. The editor is a simple graphical user interface (GUI) application for creating and editing ISTP/SPDF guideline-compliant skeleton CDF files. The SPDF ISTP CDF skeleton editor consists of the following components: A swing-based Java GUI program, JavaHelp-based manual/ tutorial, Image/Icon files, and HTML Web page for distribution. The editor is available as a traditional Java desktop application as well as a Java Network Launching Protocol (JNLP) application. Once started, it functions like a typical Java GUI file editor application for creating/editing application-unique files.

Chimiak, Reine↗

The Future of NASA Earth Science in the Commercial Cloud: Challenges and Opportunities

NASA produces a large volume and variety of data products that are used every day to support research, decision making, and education. The widespread use of NASA’s Earth Science data is enabled by NASA’s Earth Science Data System (ESDS) program, which oversees the archiving and distribution of these data and invests in the development of new data systems and tools. However, NASA’s current approach to Earth Science data distribution — based on distributed institutional archives with individual on-premises high-performance computing capabilities — faces some significant challenges, including massive increases in data volume from upcoming missions, a greater need for transdisciplinary science that synthesizes many different kinds of observations, and a push to make science more open, inclusive, and accessible. To address these challenges, NASA is aggressively migrating its Earth Science data and related tools and services into the commercial cloud. Migration of data into the commercial cloud can significantly improve NASA’s existing data system capabilities by (1) providing more flexible options for storage and compute (including rapid, as-needed access to state-of-the-art capabilities); (2) by centralizing and standardizing data access, which gives all of NASA’s institutional data centers access to all of each other’s datasets; and (3) by facilitating “analysis-in-place”, whereby users can bring their own computational workflows and tools to the data rather than having to maintain their own copies of NASA datasets. However, migration to the commercial cloud also poses some significant challenges, including (1) managing costs under a “pay-as-you-go” model; (2) incompatibility with existing tools and data formats with object-based storage and network access; (3) vendor lock-in; (4) challenges with data access for workflows that mix on-premise and cloud computing; and (5) standardization for highly diverse data as is present in NASA’s data archive. I conclude with two examples of recent NASA activities showcasing capabilities enabled by the commercial cloud: An interactive analysis and development platform for analyzing airborne imaging spectroscopy data, and a new collection of tools and services for data discovery, analysis, publication, and data-driven storytelling (Visualization, Exploration, and Data Analysis, VEDA).

Alexey N Shiklomanov↗

Power-oriented bus encoding for data transmission

Systems, apparatuses, and methods for reducing the toggle rates on buses are disclosed. A computing system includes a source which provides packets for transmission on a bus. The packet is compressed by a compression engine. The compressed data format of the packet includes locations (bit positions) referred to as holes which do not include valid data. A bus configuration module identifies the locations of the holes and replaces the holes with information from a previous packet transmitted earlier on the bus. The bus configuration module also determines a new transmission bus width for the packet for lowering the bus toggle rate on the bus during transmission.

42 ENGINEERING↗