Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Metadata”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20

Semantics-based distributed I/O with the ParaMEDIC framework.

Many large-scale applications simultaneously rely on multiple resources for efficient execution. For example, such applications may require both large compute and storage resources; however, very few supercomputing centers can provide large quantities of both. Thus, data generated at the compute site oftentimes has to be moved to a remote storage site for either storage or visualization and analysis. Clearly, this is not an efficient model, especially when the two sites are distributed over a wide-area network. Thus, we present a framework called 'ParaMEDIC: Parallel Metadata Environment for Distributed I/O and Computing' which uses application-specific semantic information to convert the generated data to orders-of-magnitude smaller metadata at the compute site, transfer the metadata to the storage site, and re-process the metadata at the storage site to regenerate the output. Specifically, ParaMEDIC trades a small amount of additional computation (in the form of data post-processing) for a potentially significant reduction in data that needs to be transferred in distributed environments.

99 GENERAL AND MISCELLANEOUS↗

Scalable Incremental Checkpointing using GPU-Accelerated De-Duplication

Writing large amounts of data concurrently to stable storage is a typical I/O pattern of many HPC workflows. This pattern introduces high I/O overheads and results in increased storage space utilization especially for workflows that need to capture the evolution of data structures with high frequency as checkpoints. In this context, many applications, such as graph pattern matching, perform sparse updates to large data structures between checkpoints. For these applications, incremental checkpointing techniques that save only the differences from one checkpoint to another can dramatically reduce the checkpoint sizes, I/O bottlenecks, and storage space utilization. However, such techniques are not without challenges: it is non-trivial to transparently determine what data has changed since a previous checkpoint and assemble the differences in a compact fashion that does not result in excessive metadata. State-of-art data reduction techniques (e.g., compression and de-duplication) have significant limitations when applied to modern HPC applications that leverage GPUs: slow at detecting the differences, generate a large amount of metadata to keep track of the differences, and ignore crucial spatiotemporal checkpoint data redundancy. This paper addresses these challenges by proposing a Merkle tree-based incremental checkpointing method to exploit GPUs' high memory bandwidth and massive parallelism. Experimental results at scale show a significant reduction of the I/O overhead and space utilization of checkpointing compared with state-of-the-art incremental checkpointing and compression techniques.

Tan, Nigel↗

MIQS v0.6

In this research, we propose a novel Metadata Indexing and Querying for Self-describing formats (MIQS), which removes the external DBMS and utilizes in-memory index to achieve efficient metadata searching. MIQS follows the self-contained data management paradigm and provides portable and schema-free metadata indexing and querying functionalities for self-describing file formats. We have evaluated MIQS with the state-of-the-art MongoDB-based metadata indexing solution. MIQS achieved up to 99% time reduction in index construction and up to 172,000X search performance improvement with up to 75% reduction in memory footprint.

Byna, Suren↗

MIQS v0.6

In this research, we propose a novel Metadata Indexing and Querying for Self-describing formats (MIQS), which removes the external DBMS and utilizes in-memory index to achieve efficient metadata searching. MIQS follows the self-contained data management paradigm and provides portable and schema-free metadata indexing and querying functionalities for self-describing file formats. We have evaluated MIQS with the state-of-the-art MongoDB-based metadata indexing solution. MIQS achieved up to 99% time reduction in index construction and up to 172,000X search performance improvement with up to 75% reduction in memory footprint.

Byna, Suren↗

Tropospheric Ozone Assessment Report: Database and Metrics Data of Global Surface Ozone Observations

In support of the first Tropospheric Ozone Assessment Report (TOAR) a relational database of global surface ozone observations has been developed and populated with hourly measurement data and enhanced metadata. A comprehensive suite of ozone data products including standard statistics, health and vegetation impact metrics, and trend information, are made available through a common data portal and a web interface. These data form the basis of the TOAR analyses focusing on human health, vegetation, and climate relevant ozone issues, which are part of this special feature. Cooperation among many data centers and individual researchers worldwide made it possible to build the world's largest collection of in-situ hourly surface ozone data covering the period from 1970 to 2015. By combining the data from almost 10,000 measurement sites around the world with global metadata information, new analyses of surface ozone have become possible, such as the first globally consistent characterisations of measurement sites as either urban or rural/remote. Exploitation of these global metadata allows for new insights into the global distribution, and seasonal and long-term changes of tropospheric ozone and they enable TOAR to perform the first, globally consistent analysis of present-day ozone concentrations and recent ozone changes with relevance to health, agriculture, and climate. Considerable effort was made to harmonize and synthesize data formats and metadata information from various networks and individual data submissions. Extensive quality control was applied to identify questionable and erroneous data, including changes in apparent instrument offsets or calibrations. Such data were excluded from TOAR data products. Limitations of a posteriori data quality assurance are discussed. As a result of the work presented here, global coverage of surface ozone data for scientific analysis has been significantly extended. Yet, large gaps remain in the surface observation network both in terms of regions without monitoring, and in terms of regions that have monitoring programs but no public access to the data archive. Therefore future improvements to the database will require not only improved data harmonization, but also expanded data sharing and increased monitoring in data-sparse regions.

54 ENVIRONMENTAL SCIENCES↗

ESS-DIVE Unoccupied Aerial Systems (UAS) Reporting Format v1

Here we present documentation of the ESS-DIVE reporting format for Unoccupied Aerial System (UAS) data and metadata. This reporting format provides guidance to data contributors on how to store data to maximize their discoverability, facilitate their efficient reuse, and add value to individual datasets. For data users, the reporting format will better allow data repositories to optimize data search and extraction, and more readily integrate similar data into harmonized synthesis products. The reporting format provides templates and guidance for the reporting of metadata for UAS experimental campaigns, individual flights, platform and sensor description. To improve data access and discoverability, the reporting format proposes a data description scheme of Levels based on the degree of processing, where Level 0 includes raw data, through to Level 3 being derived data end products. A range of examples of data types for each Level are given, with suggested file naming schemes. The reporting format presented here is intended to form a foundation for future development that will accommodate new UAS technologies and approaches to data access and use in the future. The reporting format documentation is maintained and updated on the ESS-DIVE Community Space GitHub at https://github.com/ess-dive-community/essdive-uas. This data package is the first published version of this reporting format, and comprises a zip file of the complete content of https://github.com/ess-dive-community/essdive-uas v1.0. The zip contains the reporting format description, instructions and variable definitions in GitHub markdown language (*.md) and metadata templates in csv format. The reporting format is designed to be compatible with other ESS-DIVE formats, and it is specifically recommended that this reporting format be used in conjunction with the File-level metadata (FLMD) and comma separated values (csv) reporting formats for submission to the ESS-DIVE repository.

54 ENVIRONMENTAL SCIENCES↗

Leaf water potential, Feb2016-May2016, PA-SLZ, PA-PNM, PA-BCI: Panama

This data package contains leaf water potential data from the Barro Colorado Island (BCI), Parque Natural Metropolitano (PNM), and Bosque Protector San Lorenzo (SLZ) NGEE Tropics field sites in Panama. Pre-dawn and diurnal leaf water potential were measured on a monthly basis from February to May 2016 at SLZ and PNM. Data from BCI are only available for the month of March. This data was collected as part of the 2016 El Niño-Southern Oscillation (ENSO) campaign. Included in the attached zip file are data and metadata folders. The single data file "2016ENSO_Panama_LWP" has been provided in both Excel and CSV formats for usability purposes. The metadata file "Metadata_description_2016_ENSO_Panama" provides protocols, site descriptions, equipment information, and more, and has been provided in .docx and PDF file formats. See related datasets (existing and future) for further sample details, leaf spectra, leaf mass area (LMA), gas exchange and leaf chemistry data. VERSION 2 update. The identification of a species from the PNM site has been corrected as follows: the identification of the tree initially identified as Pseudosamanea guachapele (ALBIED) has been revised to Albizia adinocephala (ALBIAD). The updated data package includes revised data, metadata and protocol documents updated to reflect this change.

54 ENVIRONMENTAL SCIENCES↗

Topological Analysis of The SPOKE Graph

The SPOKE graph [2, 6] is a sparse decorated semantic graph representing a collection of knowledge collected in many scientific databases from the fields of healthcare, biochemistry, chemistry, biology, et cetera. This knowledge graph is stored as a relational dataset decorated with metadata on each constituent vertex and edge. Formally, the graph is G(V, E, D), where V is a set of n vertices V := {1, ..., n} and edges of the form (i, j) ϵ E for i, j ϵ V, and table D that for any item in V υ E stores unstructured data such as vertex/edge type, nature of a relationship, et cetera. D(i) = {data involving vertex i ϵ V}, and D(i, j) = {data involving edge (i, j) ϵ E}. Here, we treat the graph as undirected in the sense that a direct relationship for (i, j) causes a (possibly opposite) reverse direct relationship for (j, i). The SPOKE graph G(V, E, D) is formed by processing a collection of relational datasets from medicine, chemistry, and biology, connecting many entities. Here, we analyze an instance from 2019, Spoke-20190707, where a graph file contains 6.16M edges and associated metadata and a vertex file contains 2.15M vertices and the associated metadata. There are 12 different types of vertex entities; all edge types used are implicit (see §2). There is other metadata in D on edges and vertices, but we just use the topology and the vertex labels in this report. SPOKE is growing as more knowledge is gained and more datasets are added. SPOKE is likely to grow 10x during the next phase of this project, and we therefore would like to consider topoligical analysis techniques that are scalable to several orders of magnitude larger than the current dataset (say >1B edges).

59 BASIC BIOLOGICAL SCIENCES↗

DOE FAIR Surrogate Benchmarks Supporting AI and Simulation Research (SBI Surrogate Benchmark Initiative) (Final Report)

Computational Science is being revolutionized by integrating AI and simulation and, in particular, by deep learning surrogate models that can replace all or part of traditional large‐scale HPC computations. Such surrogates can achieve remarkable performance improvements, as much as several orders of magnitude, and save both compute time and energy. The Surrogate Benchmark Initiative (SBI) project creates a community repository and FAIR (Findable, Accessible, Interoperable, and Reusable) data ecosystem for HPC application surrogate benchmarks. The SBI team comes from Argonne National Laboratory (ANL), Indiana University (IU), Rutgers University, the University of Tennessee, Knoxville (UTK), and the University of Virginia(UVA). SBI repositories include data, code, and all relevant collateral artifacts, that the science and engineering community needs to use and reuse these data sets and surrogates. SBI repositories generate active research from both participants in SBI and the broader AI and domain science communities. This project develops surrogates that use several different neural nets to learn and quickly infer the results of simulations and data systems and capture them as surrogate benchmarks with a rich set of metadata, covering. Data; Model; Metrics specification; Machine specification; Science, Speed, Power Results, We research FAIR metadata for these benchmarks. We develop application surrogate examples as benchmarks across many fields (ANL, UTK, IU, UVA). We also study non Surrogate benchmarks that have many common features and similar issues regarding FAIRness. We work with MLCommons (UVA, UTK), which is a major machine learning benchmarking activity where we get metadata ontologies, software, and benchmarks, benchmarks have datasets, models, and metadata, and they need a technical framework developed by UTK and Rutgers and deployed by UVA. We study features of Surrogates, including performance, training set size, and uncertainty quantification (Rutgers, UVA and IU).

97 MATHEMATICS AND COMPUTING↗

FAIR Surrogate Benchmarks Supporting AI and Simulation Research (Final Report)

Computational Science is being revolutionized by integrating AI and simulation and, in particular, by deep learning surrogate models that can replace all or part of traditional large‐scale HPC computations. Such surrogates can achieve remarkable performance improvements, as much as several orders of magnitude, and save both compute time and energy. The Surrogate Benchmark Initiative (SBI) project creates a community repository and FAIR (Findable, Accessible, Interoperable, and Reusable) data ecosystem for HPC application surrogate benchmarks. The SBI team comes from Argonne National Laboratory (ANL), Indiana University (IU), Rutgers University, the University of Tennessee, Knoxville (UTK), and the University of Virginia (UVA). SBI repositories include data, code, and all relevant collateral artifacts that the science and engineering community need to use and reuse these data sets and surrogates. SBI repositories generate active research from both the participants in SBI and the broad community of AI and domain scientists. This project develops surrogates that use several different neural nets to learn and quickly infer the results of simulations and data systems and captures them as surrogate benchmarks with a rich set of metadata covering: Data; Model; Metrics specification; Machine specification; and Science, Speed, and Power Results. We research FAIR metadata for these benchmarks. We develop application surrogate examples as benchmarks across many fields (ANL, UTK, IU, UVA). We also study non-Surrogate benchmarks that have many common features and similar issues as regards FAIRness. We work with MLCommons (UVA, UTK), which is a major machine learning benchmarking activity where we get metadata ontologies, software, and benchmarks, Benchmarks have datasets, models, and metadata and they need a technical framework developed by UTK and Rutgers and deployed by UVA. We study features of Surrogates including performance, training set size, and uncertainty quantification (Rutgers, UVA and IU).

97 MATHEMATICS AND COMPUTING↗

PRIDE Surveillance Projects Data Packaging Project, Information Package Specification Version 1.0

This document contains a specification for a standard XML document format called an information package that can be used to store information and the context required to understand and use that information in information management systems and other types of information archives. An information package consists of packaged information, a set of information metadata that describes the packaged information, and an XML signature that protects the packaged information. The information package described in this specification was designed to be used to store Department of Energy (DOE) and National Nuclear Security Administration (NNSA) information and includes the metadata required for that information: a unique package identifier, information marking that conforms to DOE and NNSA requirements, and access control metadata. Information package metadata can also include information search terms, package history, and notes. Packaged information can be text content, binary content, and the contents of files and other containers. A single information package can contain multiple types of information. All content not in a text form compatible with XML must be in a text encoding such as base64. Package information is protected by a digital XML signature that can be used to determine whether the information has changed since it was signed and to identify the source of the information. This specification has been tested but has not been used to create production information packages. The authors expect that gaps and unclear requirements in this specification will be identified as this specification is used to create information packages and as information stored in information packages is used. The authors expect to issue revised versions of this specification as needed to address these issues.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

HydroFish: freshwater fish co-occurrence with hydropower plants and non-powered dams in conterminous United States sub-basins

The HydroFish dataset lists all existing hydropower plants (EHAs) and non-powered dams (NPDs; ≥ 0.001 MW potential nominal capacity), delineates the hydrologic sub-basins in which they are situated, and then lists all freshwater fish species reported to occur in those sub-basins. This dataset was compiled using the HydroBio dataset (https://hydrosource.ornl.gov/data/datasets/hydrobio/) and contains 24 total variables that describe hydrologic sub-basins, each unique EHA (plant ID and name, geographic coordinates, permit type, capacity, etc.) and NPD (ID value, known names, geographic coordinates, and estimated potential nominal capacity), and freshwater fish species in the sub-basin (common and scientific name, origin, and migratory and threat status). The HydroBio dataset was built using Oak Ridge National Laboratory’s Existing Hydropower Assets Dataset (2024 version) and Non-powered Dam Technical Potential Dataset (2024 version), and NatureServe’s fish species distribution dataset (2023 version). The dataset also contains summary variables that report the unique number of EHAs, NPDs, and freshwater fish species per sub-basin. The HydroFish dataset contains two unique data files: 1) a .csv metadata file describing the dataset variables, and 2) a .csv data file containing the actual dataset. Note that there may be many rows per unique existing hydropower plant or non-powered dam given that distinct species are listed per existing plant or NPD per sub-basin. The dataset is downloadable as a zip file containing the metadata and dataset files.

Bozeman, Bryan [Oak Ridge National Laboratory (ORN↗

Epi Archive: Automated Synthesis of Global Notifiable Disease Data

ObjectiveLANL has built software that automatically collects global notifiable disease data, synthesizes the data, and makes it available to humans and computers within the Biosurveillance Ecosystem (BSVE) as a novel data stream. These data have many applications including improving the prediction and early warning of disease events.IntroductionMost countries do not report national notifiable disease data in a machine-readable format. Data are often in the form of a file that contains text, tables and graphs summarizing weekly or monthly disease counts. This presents a problem when information is needed for more data intensive approaches to epidemiology, biosurveillance and public health.While most nations likely store incident data in a machine-readable format, governments are often hesitant to share data openly for a variety of reasons that include technical, political, economic, and motivational issues1.A survey conducted by LANL of notifiable disease data reporting in over fifty countries identified only a few websites that report data in a machine-readable format. The majority (>70%) produce reports as PDF files on a regular basis. The bulk of the PDF reports present data in a structured tabular format, while some report in natural language.The structure and format of PDF reports change often; this adds to the complexity of identifying and parsing the desired data. Not all websites publish in English, and it is common to find typos and clerical errors.LANL has developed a tool, Epi Archive, to collect global notifiable disease data automatically and continuously and make it uniform and readily accessible.MethodsWe conducted a survey of the national notifiable disease reporting systems notating how the data are reported and in what formats. We determined the minimal metadata that is required to contextualize incident counts properly, as well as optional metadata that is commonly found.The development of software to regularly ingest notifiable disease data and make it available involves three or four main steps: scraping, detecting, parsing and persisting.Scraping: we examine website design and determine reporting mechanisms for each country/website, as well as what varies across the reporting mechanisms. We then designed and wrote code to automate the downloading of the data for each country. We store all artifacts presented as files (PDF, XLSX, etc.) in their original form, along with appropriate metadata for parsing and data provenance.Detecting: This step is required when parsing structured non-machine-readable data such as tabular data in PDF files. We combined the Nurminen methodology of PDF table detection with in-house heuristics to find the desired data within PDF reports2.Parsing: We determined what to extract from each dataset and parsed these data into uniform data structures, correctly accommodating the variations in metadata (e.g., time interval definitions) and the various human languages.Persisting: We store the data in the Epi Archive database and make it available on the internet and through the BSVE. The data is persisted into a structured and normalized SQL database.ResultsThe Epi Archive tool currently contains national and/or subnational notifiable disease data from twenty nations. When a user accesses the Epi Archive site, they are prompted with four fields: country, subregion, disease of interest, and date duration. Upon form submission, a time series is generated from the users’ specifications. The generated graph can then be downloaded into a CSV file if a user is interested in performing personal analysis. Additionally, the data from Epi Archive can be reached through a REST API (Representational State Transfer Application Programming Interface).ConclusionsLANL, as part of a currently funded DTRA effort, is automatically and continually collecting global notifiable disease data. While 20 nations are in production, more are being brought online in the near future. These data are already being utilized and will have many applications including improving the prediction and early warning of disease events.References[1] van Panhuis WG, Paul P, Emerson C, et al. A systematic review of barriers to data sharing in public health. BMC Public Health. 2014. 14:1144. doi:10.1186/1471-2458-14-1144[2] Nurminen, Anssi. "Algorithmic extraction of data in tables in PDF documents." (2013).

Khalsa, Hari S.↗

Projected Urban Morphology of the Los Angeles Area by the Year 2100

This dataset provides projections of urban building morphologies for the Los Angeles urban area at 30-meter spatial resolution. It contains 192 raster files that detail two primary building attributes: building footprint fractions (ranging from 0 to 1) and average building heights (ranging from 0 to 75 meters). The projections account for a wide range of future pathways, covering two Shared Socioeconomic Pathway (SSP) scenarios (SSP3 and SSP5), two population scenarios, two developed land intensification scenarios, and four distinct levels of intensification. The dataset was created using dual Generative Adversarial Networks (GANs) trained on 2015 land cover and building properties from the National Land Cover Database (NLCD) and Model America datasets. Supporting information on the dataset has been described in the LAUrbanAreaMorphologyProjections2100_README.txt file.

Pandey, Bhartendu↗

The health care and life sciences community profile for dataset descriptions

Access to consistent, high-quality metadata is critical to finding, understanding, and reusing scientific data. However, while there are many relevant vocabularies for the annotation of a dataset, none sufficiently captures all the necessary metadata. This prevents uniform indexing and querying of dataset repositories. Towards providing a practical guide for producing a high quality description of biomedical datasets, the W3C Semantic Web for Health Care and the Life Sciences Interest Group (HCLSIG) identified Resource Description Framework (RDF) vocabularies that could be used to specify common metadata elements and their value sets. The resulting guideline covers elements of description, identification, attribution, versioning, provenance, and content summarization. This guideline reuses existing vocabularies, and is intended to meet key functional requirements including indexing, discovery, exchange, query, and retrieval of datasets, thereby enabling the publication of FAIR data. The resulting metadata profile is generic and could be used by other domains with an interest in providing machine readable descriptions of versioned datasets.

Science & Technology - Other Topics↗

Analysis of Gaps in Geothermal Exploration Data in the Western United States

The USGS Resource Assessment (Williams et al. 2008) outlined a mean 30 GW, of Undiscovered Hydrothermal Resource in the western U.S. One goal of the Geothermal Technology Office (GTO) is to accelerate the development of this undiscovered resource. To help guide its future data collection efforts, GTO funded a data gap analysis in FY2012 to identify high potential hydrothermal areas where critical data are needed. For the gap analysis, information was collected about existing data and data coverage maps were created with this information. Next, these maps were compared with the USGS resource potential maps to identify priority sites for exploration data collection. Though there are many types of data that can be used for hydrothermal exploration, five types of exploration data were targeted for this analysis. These data types were elected for their regional reconnaissance potential, and include many of the primary exploration echniques currently used by the geothermal industry. The data types include: 1) well data; 2) geologic maps; 3) fault maps; 4) geochemistry data; and 5) geophysical data. To determine data coverage, metadata for exploration data (including data type, data status, and coverage information) were collected and catalogued. Many industry, university, laboratory, and agency personnel were contacted in order to collect metadata for this effort. Data and metadata provided by the contributors to the National Geothermal Data System (NGDS) were also incorporated in the gap analysis. An online tool was developed to allow all geothermal data providers to directly add metadata themselves and view the results of the analys is via maps of data coverage in Geothermal Prospector (http://maps.nrel.gov/gt_prospector). A grid of the contiguous U.S. was created with 88,000 10-km by 10-km grid cells, and each cell was populated with the status of data availability corresponding to the five data types. Using these five data coverage maps and the USGS Resource Potential Map, sites were identified for future data collection efforts. These sites signify both that the USGS has indicated high favorability of occurrence of geothermal resources and that data gaps exist.

15 GEOTHERMAL ENERGY↗

Data compression system using base values and methods thereof

In some embodiments, a memory controller in a processor includes a base value cache, a compressor, and a metadata cache. The compressor is coupled to the base value cache and the metadata cache. The compressor compresses a data block using at least a base value and delta values. The compressor determines whether the size of the data block exceeds a data block threshold value. Based on the determination of whether the size of the compressed data block generated by the compressor exceeds the data block threshold value, the memory controller transfers only a set of the compressed delta values to memory for storage. A decompressor located in the lower level cache of the processor decompresses the compressed data block using the base value stored in the base value cache, metadata stored in the metadata cache and the delta values stored in memory.

Seyedzadehdelcheh, SeyedMohammad↗

Data compression system using base values and methods thereof

In some embodiments, a memory controller in a processor includes a base value cache, a compressor, and a metadata cache. The compressor is coupled to the base value cache and the metadata cache. The compressor compresses a data block using at least a base value and delta values. The compressor determines whether the size of the data block exceeds a data block threshold value. Based on the determination of whether the size of the compressed data block generated by the compressor exceeds the data block threshold value, the memory controller transfers only a set of the compressed delta values to memory for storage. A decompressor located in the lower level cache of the processor decompresses the compressed data block using the base value stored in the base value cache, metadata stored in the metadata cache and the delta values stored in memory.

Seyedzadehdelcheh, Seyed Mohammad↗