Metadata Classification Machine Learning Data
This GitLab project contains the training data that was used for the metadata machine learning classification project.
54 ENVIRONMENTAL SCIENCES↗
SEARCH · Engineering Papers
Results for “metadata”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Big Earth Data Initiative: Metadata Improvement: Case Studies
Big Earth Data Initiative (BEDI) The Big Earth Data Initiative (BEDI) invests in standardizing and optimizing the collection, management and delivery of U.S. Government's civil Earth observation data to improve discovery, access use, and understanding of Earth observations by the broader user community. Complete and consistent standard metadata helps address all three goals.
BEDI↗
ES2Vec: Earth Science Metadata Suggestions and Analogical Reasoning
As the volume of text-based Earth science research grows, it is increasingly possible to discover latent relationships in the literature. However, traditional methodologies are restricted by limited computational capabilities and intractable problem spaces. Advancements in natural language processing (NLP) have allowed us to use an extensive Earth science corpus to create a domain-specific word vector model, Es2Vec, which we have used to surface latent relationships between Earth science concepts and generate improved keyword tags. Earth science metadata keyword assignment is a challenging problem. Dataset curators select appropriate keywords from the Global Change Master Directory (GCMD) set of keywords. The keywords an are integral part of the search and discovery of these datasets. Hence, the selection of keywords is crucial to increasing the discoverability of datasets. Utilizing machine learning techniques, we provide users with automated keyword suggestions to complement manual selection. We trained a machine learning model that leverages the semantic embedding ability of Word2Vec models to process abstracts and suggest relevant keywords. A user interface tool we built to assist data curators in the assignment of such keywords is also described.
word vectors↗
Development of an Improved Spatial Metadata Simplification Algorithm
The National Aeronautics and Space Administration's (NASA) Atmospheric Science Data Center (ASDC) at NASA Langley Research Center in Hampton, VA provides atmospheric science data products and services to the science community, including enhanced search and subsetting capabilities for numerous Earth Science datasets. The ASDC is the official Distributed Active Archive Center (DAAC) of record for the Tropospheric Emissions: Monitoring of Pollution (TEMPO) instrument. TEMPO is situated on a geostationary satellite positioned at a longitude near the center of the conterminous United States and focused on North America, making hourly swaths of its field of regard from east to west. Spatial metadata is an essential component for the discovery and distribution of Earth Science data. The simplified polygonal boundaries representing the archived data files ensure that any granule can be identified quickly and accurately by a geospatial query. Historically the Douglas-Peucker algorithm has been used for polygon simplification; however, due to the nature of the algorithm, a buffer must be added to the polygon before simplification to ensure pivotal points are not removed by the algorithm. This adds in additional error to the polygon simplification. ASDC’s goal is to test other methods of polyline simplification, such as Visvalingan-Whyatt and Opheim simplification alongside of Douglas-Peucker and different buffering methods, to produce less error during polygon simplification of TEMPO data swaths, and special spatial query geometries such as EPA non-attainment regions, and geopolitical boundaries.
Spatial Metadata↗
Use of Spatial Metadata Simplification for TEMPO
The National Aeronautics and Space Administration's (NASA) Atmospheric Science Data Center (ASDC) at NASA Langley Research Center in Hampton, VA provides atmospheric science data products and services to the science community, including enhanced search and subsetting capabilities for numerous Earth Science datasets. The ASDC is the official Distributed Active Archive Center (DAAC) of record for the Tropospheric Emissions: Monitoring of Pollution (TEMPO) instrument. TEMPO is situated on a geostationary satellite positioned at a longitude near the center of the conterminous United States and focused on North America, making hourly swaths of its field of regard from east to west. Spatial metadata is an essential component for the discovery and distribution of Earth Science data. The simplified polygonal boundaries representing the archived data files ensure that any granule can be identified quickly and accurately by a geospatial query. Historically the Douglas-Peucker algorithm has been used for polygon simplification; however, due to the nature of the algorithm, a buffer must be added to the polygon before simplification to ensure pivotal points are not removed by the algorithm. This adds in additional error to the polygon simplification. ASDC’s goal is to test other methods of polyline simplification, such as Visvalingan-Whyatt and Opheim simplification alongside of Douglas-Peucker and different buffering methods, to produce less error during polygon simplification of TEMPO data swaths, and special spatial query geometries such as EPA non-attainment regions, and geopolitical boundaries.
Spatial Metadata↗
EXTRACT: Interactive extraction of environment metadata and term suggestion for metagenomic sample annotation
The microbial and molecular ecology research communities have made substantial progress on developing standards for annotating samples with environment metadata. However, sample manual annotation is a highly labor intensive process and requires familiarity with the terminologies used. We have therefore developed an interactive annotation tool, EXTRACT, which helps curators identify and extract standard-compliant terms for annotation of metagenomic records and other samples. Behind its web-based user interface, the system combines published methods for named entity recognition of environment, organism, tissue and disease terms. The evaluators in the BioCreative V Interactive Annotation Task found the system to be intuitive, useful, well documented and sufficiently accurate to be helpful in spotting relevant text passages and extracting organism and environment terms. Here the comparison of fully manual and text-mining-assisted curation revealed that EXTRACT speeds up annotation by 15–25% and helps curators to detect terms that would otherwise have been missed.
59 BASIC BIOLOGICAL SCIENCES↗
IMG/VR v4: an expanded database of uncultivated virus genomes within a framework of extensive functional, taxonomic, and ecological metadata
Viruses are widely recognized as critical members of all microbiomes. Metagenomics enables large-scale exploration of the global virosphere, progressively revealing the extensive genomic diversity of viruses on Earth and highlighting the myriad of ways by which viruses impact biological processes. IMG/VR provides access to the largest collection of viral sequences obtained from (meta)genomes, along with functional annotation and rich metadata. A web interface enables users to efficiently browse and search viruses based on genome features and/or sequence similarity. Here, for this work, we present the fourth version of IMG/VR, composed of >15 million virus genomes and genome fragments, a ≈6-fold increase in size compared to the previous version. These clustered into 8.7 million viral operational taxonomic units, including 231 408 with at least one high-quality representative. Viral sequences in IMG/VR are now systematically identified from genomes, metagenomes, and metatranscriptomes using a new detection approach (geNomad), and IMG standard annotation are complemented with genome quality estimation using CheckV, taxonomic classification reflecting the latest taxonomic standards, and microbial host taxonomy prediction. IMG/VR v4 is available at https://img.jgi.doe.gov/vr, and the underlying data are available to download at https://genome.jgi.doe.gov/portal/IMG_VR.
59 BASIC BIOLOGICAL SCIENCES↗
Raw diffraction data preservation and reuse: Overview, update on practicalities and metadata requirements
A topical review is presented of the rapidly developing interest in and storage options for the preservation and reuse of raw data within the scientific domain of the IUCr and its Commissions, each of which operates within a great diversity of instrumentation. A résumé is included of the case for raw diffraction data deposition. An overall context is set by highlighting the initiatives of science policy makers towards an `Open Science' model within which crystallographers will increasingly work in the future; this will bring new funding opportunities but also new codes of procedure within open science frameworks. Skills education and training for crystallographers will need to be expanded. Overall, there are now the means and the organization for the preservation of raw crystallographic diffraction data via different types of archive, such as at universities, discipline-specific repositories (Integrated Resource for Reproducibility in Macromolecular Crystallography, Structural Biology Data Grid), general public data repositories (Zenodo, ResearchGate) and centralized neutron and X-ray facilities. Formulation of improved metadata descriptors for the raw data types of each of the IUCr Commissions is in progress; some detailed examples are provided. Lastly, a number of specific case studies are presented, including an example research thread that provides complete open access to raw data.
96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
Insufficient reporting of x-ray photoelectron spectroscopy instrumental and peak fitting parameters (metadata) in the scientific literature
This study was motivated by earlier observations. It is a systematic examination of the adequacy of reporting of information (metadata) necessary to understand x-ray photoelectron spectroscopy (XPS) data collection and data analysis in the scientific literature. The information for this study was obtained from papers published in three high-quality journals over a six-month period in 2019 and throughout 2021. Each paper was evaluated to determine whether the authors had reported (percentages of the papers properly providing the information are given in parentheses) the spectrometer (66%), fitting software (15%), x-ray source (40%), pass energy (10%), spot size (5%), synthetic peak shapes in fits (10%), backgrounds in fits (10%), whether the XPS data are shown in the main body of the paper or in the supporting information (or both), and whether fitted or unfitted spectra were shown (80% of published spectra are fit). The Shirley background is the most widely used background in XPS peak fitting. The Al Kα source is the most widely used x-ray source for XPS data collection. CASAXPS is the most widely used fitting program for XPS data analysis. Further, there is good agreement between the results gathered during the two years of our survey. There are some hints the situation may be improving. This study also provides a list of the information/parameters that should be reported when XPS is performed.
47 OTHER INSTRUMENTATION↗
Scalable Pattern Matching in Metadata Graphs via Constraint Checking
Pattern matching is a fundamental tool for answering complex graph queries. Unfortunately, existing solutions have limited capabilities: They do not scale to process large graphs and/or support only a restricted set of search templates or usage scenarios. Moreover, the algorithms at the core of the existing techniques are not suitable for today’s graph processing infrastructures relying on horizontal scalability and shared-nothing clusters, as most of these algorithms are inherently sequential and difficult to parallelize. In this article we present an algorithmic pipeline that bases pattern matching on constraint checking. The key intuition is that each vertex and edge participating in a match has to meet a set of constraints implicitly specified by the search template. These constraints can be verified independently and typically are less expensive to compute than searching the full template. The pipeline we propose generates these constraints and iterates over them to eliminate all the vertices and edges that do not participate in any match, thus reducing the background graph to a subgraph that is the union of all template matches—the complete set of all vertices and edges that participate in at least one match. Additional analysis can be performed on this annotated, reduced graph, such as full match enumeration, match counting, or computing vertex/edge centrality. Furthermore, a vertex-centric formulation for constraint checking algorithms exists, and this makes it possible to harness existing high-performance, vertex-centric graph processing frameworks. This technique (i) enables highly scalable pattern matching in metadata (labeled) graphs; (ii) supports arbitrary patterns with 100% precision; (iii) enables tradeoffs between precision and time-to-solution, while always selects all vertices and edges that participate in matches, thus offering 100% recall; and (iv) supports a set of popular data analytics scenarios. We implement our approach on top of HavoqGT, an open-source asynchronous graph processing framework, and demonstrate its advantages through strong and weak scaling experiments on massive scale real-world (up to 257 billion edges) and synthetic (up to 4.4 trillion edges) labeled graphs, respectively, and at scales (1,024 nodes / 36,864 cores), orders of magnitude larger than used in the past for similar problems. This article serves two purposes: First, it synthesises the knowledge accumulated during a long-term project. Second, it presents new system features, usage scenarios, optimizations, and comparisons with related work that strengthen the confidence that pattern matching based on iterative pruning via constraint checking is an effective and scalable approach in practice. The new contributions include the following: (i) We demonstrate the ability of the constraint checking approach to efficiently support two additional search scenarios that often emerge in practice, interactive incremental search and exploratory search. (ii) We empirically compare our solution with two additional state-of-the-art systems, Arabsque and TriAD. (iii) We show the ability of our solution to accommodate a more diverse range of datasets with varying properties, e.g., scale, skewness, label distribution, and match frequency. (iv) We introduce or extend a number of system features (e.g., work aggregation, load balancing, and the ability to cap the generated traffic) and design optimizations and demonstrate their advantages with respect to improving performance and scalability. (v) We present bottleneck analysis and insights into artifacts that influence performance. (vi) We present a theoretical complexity argument that motivates the performance gains we observe.
97 MATHEMATICS AND COMPUTING↗
Avian Activity Classification Using Recurrent Networks to Fuse Videos with Metadata on Imbalanced Datasets
Activity classification plays a crucial role in various real-life scenarios involving both humans and animals. There is an increasing need for precise activity classification focused on avian-solar interactions, as the usage of solar energy facilities, such as photovoltaic array power stations, has been observed to impact bird species richness, behavior, and activity. However, there has been no work to develop an automated system to monitor and classify these avian-solar interactions. All current methods rely on human observers, which is time and human resources costly and subject to errors related to searcher efficiency. With the recent success of Deep Learning models in activity classification problems, this paper develops a recurrent neural network-based model to automatically classify six avian activities around solar energy facilities. Our proposed model integrates critical feature engineering metadata with video frame data, enabling improved learning and more accurate activity classification. Furthermore, we address the challenge of data imbalance during training and demonstrate the efficacy of our model in detecting and classifying different activities within video tracks. Additionally, we analyze the saliency/backpropagation map of the trained proposed model and validate its decision-making rationale.
Avian activity classification; bidirectional LSTM;↗
ScienceSearch Metadata Infrastructure (ScienceSearch-MI) v1.2
ScienceSearch-MI is a framework that generates metadata from scientific artifacts (papers, proposals, images) that is used to enable search.
Ramakrishnan, Lavanya↗
Independent Metadata Update VOL Connector for HDF5 (IMM VOL) v0.1
We have built a new framework that enables independent metadata updates within HDF5 files. Our framework includes the following components: an overlay network that's optimized for 1-to-All and All-to-All communication; root-less/server-less Bcast and AllReduce operations for reduced synchronization; a distributed consensus-making platform; an execution order keeping system, and a progress engine for generic threadless concurrence handling.
Li, Tonglin↗
Locations, metadata, and species cover from field sampling survey associated with NEON AOP survey, East River, CO 2018
Locations and descriptions of the sites where field sampling was conducted during the 2018 National Ecological Observatory Network (NEON) Airborne Observation Platform (AOP) imaging spectroscopy and lidar surveys in Gunnison County, Colorado. The sampling sites were located across East River, Washington Gulch, Slate River, and Coal Creek watersheds and contained a mixture of meadow, shrub, and tree sampling sites. This data package contains the location, metadata, species composition, and vector files for the sampling sites in this project. Species vouchers that have been preserved in the Rocky Mountain Biological Laboratory herbarium can be found on the Consortia of Southern Rocky Mountain Herbaria website: http://soroherbaria.org/portal/collections/index.php. Vector files provided here are for areas delineated based on the AOP reflectance data and mosaic developed by Brodrick et al. (doi: 10.15485/1618131), and use with other datasets should be closely evaluated. Subsequent data packages will contain additional biogeochemical, microbial, and geophysical data as they become available for these sampling sites. For full documentation, please see associated reference and additional references will be added here as they become available.
2018 NEON and 2025 CHESS Campaigns↗
ESS-DIVE Reporting Format for Hydrologic Monitoring Data and Metadata
The ESS-DIVE Reporting Format (RF) for Hydrologic Monitoring Data and Metadata was designed for water level, temperature, electrical conductivity, specific conductance, dissolved oxygen, and pH measured in situ using sensors deployed in water bodies. It can also be applied to a broader suite of variables beyond its original scope. The RF is intended to provide clear and easy guidance to data generators aiming to upload consistent findable, accessible, interoperable, and reusable (FAIR) datasets. The RF includes instructions (pdf), a term guide (csv), templates (csv and xlsx), and a list of recommended vocabulary (csv) to equip the data generator with all of the tools needed to successfully apply the RF. There is also a crosswalk to some key data resources and standards (csv). Users should cite this dataset when using the RF but are encouraged to access the RF files on GitHub (https://github.com/ess-dive-community/essdive-hydrologic-monitoring) where the RF documentation is maintained and updated on the ESS-DIVE Community Space GitHub. Questions and suggestions for improvement can be submitted as GitHub issues.
54 ENVIRONMENTAL SCIENCES↗
Microbial community data from throughfall exclusion experiment: Metadata, SI, community composition, LefSe, and FunGuilR data tables from PARCHED Panama tropical forest soils, 2024-2025
Soil contains more carbon (C) than terrestrial vegetation and the atmosphere combined, with some of the largest terrestrial C stocks in tropical rainforests. Soil microbes decompose organic matter, playing a vital role in the storage or loss of soil C. With climate change, drought conditions are predicted to increase in many tropical regions, including both chronic drying and extended drought, potentially influencing these processes. This project explored the effects of chronic and seasonal drying on soil microbial communities across four distinct tropical forests in a long-term drying experiment. We investigated the effects of a chronic drying manipulation on soil microbial community abundance and variation across different forests and seasons. We also compared findings with previously published data from these forests after short-term drying. This project used soils from a long-term drying experiment established in 2018 across four seasonal lowland forests in Panama. Soils were collected from 0 – 10 cm depths during three seasonal periods in control and drying plots in 2024 and 2025 from a total of 32 plots (n = 4 per forest per treatment). The forests varied in baseline rainfall and soil fertility. We calculated alpha and beta diversity indices and compared taxonomic community composition. We found significant biogeographic variation in microbial diversity and taxonomy, with significant differences across the forests and significant effects of the drying treatment. Metadata and sample IDs are within Metadata_16S.csv and Metadata_ITS.csv. Relative abundance tables of every sample at every season are shown in the Excel workbooks 16S Relative Abundance.xlsx and ITS Relative Abundance.xlsx. They are then also shown in CSV files by each taxonomic level. Linear discriminant analysis effect size (LefSe) tables are shown for the full 16S and ITS datasets (n = 96), subsets for every site at every season (n = 8), and then for the forests with each plot merged by season (n = 8). FunGuildR data table of ITS data is uploaded.
Bacteria↗
2017 Puerto Rico G-LiHT Metadata
This data package contains metadata for data acquired for sample transects across Puerto Rico using NASA Goddard's Lidar, Hyperspectral, and Thermal (G-LiHT) Airborne Imager in March 2017. Flights specifically targeted sites with coincident field data collection for NGEE-Tropics, existing forest inventory plots from the USDA-Forest Service Forest Inventory & Analysis (FIA) program and university collaborators, and other research installations (e.g., NEON).
54 ENVIRONMENTAL SCIENCES↗
Automated metadata--final project report
This report summarizes the work of the Automated Metadata, Provenance Cataloging, and Navigable Interfaces: Ensuring the Usefulness of Extreme-Scale Data Project (MPO Project) funded by the United States Department of Energy (DOE), Offices of Advanced Scientific Computing Research and Fusion Energy Sciences. Initially funded for three years starting in 2012, it was extended for 6 months with additional funding. The project was a collaboration between scientists at General Atomics, Lawrence Berkley National Laboratory (LBNL), and Massachusetts Institute of Technology (MIT). The group leveraged existing computer science technology where possible, and extended or created new capabilities where required. The MPO project was able to successfully create a suite of software tools that can be used by a scientific community to automatically document their scientific workflows. These tools were integrated into workflows for fusion energy and climate research illustrating the general applicability of the project’s toolkit. Feedback was very positive on the project’s toolkit and the value of such automatic workflow documentation to the scientific endeavor.
96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗