SEARCH · Engineering Papers

Results for “metadata search”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

NASA NTRSDate not supplied

Keywords for All People: How Keyword Governance and Coordination with the NASA ESDIS Standards Coordination Office (ESCO) Improves GCMD Keywords for Discuvery and Use

The Global Change Master Directory (GCMD) Keywords, initiated over twenty years ago, are a hierarchical set of controlled Earth Science vocabularies that help ensure Earth science data, services, and variables are described in a consistent and comprehensive manner and allow for the precise searching of metadata and subsequent retrieval of data, services, and variables. GCMD keywords are periodically analyzed for relevancy and will continue to be refined and expanded in response to user needs. The periodic analysis is a result of successful coordination with the ESDIS Standards Coordination Office (ESCO), which is responsible for standards activities across ESDIS, and assists with providing valuable stakeholder and subject matter expert (SME) feedback on GCMD vocabularies. The ESCO is also turning to the GCMD to discuss how the keyword review process can be improved and more streamlined. In addition to the ESCO, keyword requests and feedback are also received through the GCMD Keyword Forum.

Tyler Stevens↗

NASA NTRSDate not supplied

pyQuARC: Preparing for Full Release

Metadata holds the contextual information about data and is the underlying structure for many data search portals. High quality metadata optimizes search results, allowing users to quickly retrieve the data they need. With the abundant volume and diversity of Earth observation datasets, data discovery and metadata quality are critical for end users. The Common Metadata Repository (CMR), for example, currently hosts metadata for over 9,000 Earth observation data products archived across 12 NASA Distributed Active Archive Centers (DAACs). The Analysis and Review of CMR (ARC) Team, located at Marshall Space Flight Center, assesses the completeness, correctness, and consistency of these metadata records to ensure they are accessible, usable, and discoverable. In 2021, ARC began developing pyQuARC, an open source library for Earth Observation Metadata Quality Assessment to automate this effort. The tool uses ARC’s existing metadata quality framework to provide prioritized recommendations for metadata improvement. During initial testing, pyQuARC automatically identified 58% of metadata findings when compared with a sample of manually reviewed records. Using the results from initial testing, this presentation will focus on recent advancements and improvements of the tool as the ARC team prepares for pyQuARC’s full release. It will also demonstrate pyQuARC's enrichment value, not only for the ARC team, but the broader EOSDIS metadata community as well.

Essence Raphael↗

NASA NTRSDate not supplied

Search Enhancements using Natural Language Processing Techniques

NASA Goddard Earth Sciences Data and Information Services Center (GESDISC) is one of the 12 NASA Science Mission Directorate Data Centers. The main goal of GESDISC is to provide earth science data, information, and services to the earth science data community. Consequently, data discovery is at the center of our mission and our search engine is the primary tool for our users to interact, find, and access our data. Existing search approaches are largely focused on hard-matching of keywords in the search query with dataset metadata. Here we propose to expand the search by introducing a complementary natural language processing (NLP) search. At the heart of our proposed NLP search, we trained a joint embedding using scientific text corpus and a curated set of dataset metadata. The embedding learns the association between words in our dataset metadata and those of the scientific text corpus. This enables us to go beyond simple hard-matching of a query and data set metadata and have a notion of “similarity” between the search query and the datasets. We further integrated our NLP search into the Elastic Search (ES) framework leveraging similarity search capabilities offered through the “dense_vector” field type. Our preliminary evaluations show that our proposed NLP search has the potential to be utilized to complement the existing search engine and serve as a base for a dataset recommendation system.

Armin Mehrabian↗

NASA NTRS2019

Comprehensive Assessment of Models and Events Using Library Tools (CAMEL) Framework: Time Series Comparisons

The Comprehensive Assessment of Models and Events using Library Tools (CAMEL) framework leverages existing Community Coordinated Modeling Center services: Run on Request post processing tools that generate model time series outputs and the new Community Coordinated Modeling Center Metadata Registry that describes simulation runs using Space Physics Archive Search and Extract metadata. The new CAMEL visualization tool compares the modeled time series with observational data and computes a suite of skill scores such as Prediction Efficiency, Root Mean Square Error, and Symmetric Signed Percentage Bias. Model data pairs used for skill calculations are obtained considering a user selected maximum difference between the time of observation and the nearest model output. The system renders available data for all locations and time periods selected using interactive visualizations that allow the user to zoom, pan, and pick data values along traces. Skill scores are reported for each selected event or aggregated over all events for all participating model runs. Separately, scores are reported for all locations (satellites or stations) and for each location individually. We are building on past experiences with model data comparisons of magnetosphere and ionosphere model outputs from GEM2008, GEMCEDAR Electrodynamics Thermosphere Ionosphere, and the SWPC Operational Space Weather Model challenges. The CAMEL visualization tool is demonstrated using three validation studies: (a) Wang Sheeley Arge heliosphere simulations compared against OMNI solar wind data, (b) ground magnetic perturbations from several magnetosphere and ionosphere electrodynamics models as observed by magnetometers, and (c) electron fluxes from several ring current simulations compared to Radiation Belt Storm Probes Helium Oxygen Proton Electron instrument measurements, integrated over different energy ranges.

Rastätter, Lutz↗

NASA NTRSDate not supplied

Publishing Variables Archived at GES DISC to Earth System Grid Federation (ESGF)

We present a straightforward and low-cost approach to publish variables archived at NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) to the Earth System Grid Federation (ESGF). An ESGF publication requires a single standard-name variable aggregated over time to facilitate data inter-comparison. It also contains significant metadata to enable searching in ESGF. We look up standard names on high demand in ESGF search history, and using OPeNDAP and NcML technologies we aggregate the corresponding variables available in the GES DISC archive with augmented metadata required by CMIP6 and obs4MIPs Data Specification version 2.1. At this writing 10 variables from a standard product of the Atmospheric Infrared Sounder along with the Tech Notes are published in ESGF by NASA Center for Climate Simulation (NCCS). Users can view, analyze, and subset remotely, and download these aggregated variables via links in any ESGF node after searching. We plan to work on and publish more variables and data from different NASA missions and experiments in our archive.

Fan Fang↗

DOE OSTI2026

VirJenDB: a FAIR (meta)data and bioinformatics platform for all viruses

High-throughput sequencing has generated an unprecedented volume of data. However, researcher-submitted data in repositories requires extensive curation and quality control for reuse. These tasks are hindered by the multiplicity of repositories, the sheer volume of the data, and the complexity of virus (meta)data curation. To address these challenges, VirJenDB offers a user-friendly platform to facilitate versioned, community-driven curation, and ontology development. Virus sequences were ingested from 16 sources, including ~200 fields of metadata or standards, covering taxonomy, sample, and host information. Up to 85 metadata fields have undergone at least one round of curation, and are linked to 15.4 million virus sequences, with 88 % from those infecting eukaryotes and the remaining infecting prokaryotes. Subsets were created, including a novel collection of 0.91 million viral operational taxonomic unit (vOTU) sequences across all viruses, while keeping the original sequences from each vOTU to facilitate downstream analyses, e.g. sequence variation. The VirJenDB web portal (https://www.virjendb.org) provides HTTPS and Application Programming Interface (API) access to the sequence datasets and metadata, offering a search engine, filtering, download, visualizations, and documentation. VirJenDB aims to connect the phage and eukaryotic virus research communities by supporting webtool integration, meta-analyses, and metadata schema extensions.

Saghaei, Shahram↗

DOE OSTI2024

Automated annotation of scientific texts for ML-based keyphrase extraction and validation

Advanced omics technologies and facilities generate a wealth of valuable data daily; however, the data often lack the essential metadata required for researchers to find, curate, and search them effectively. The lack of metadata poses a significant challenge in the utilization of these data sets. Machine learning (ML)–based metadata extraction techniques have emerged as a potentially viable approach to automatically annotating scientific data sets with the metadata necessary for enabling effective search. Text labeling, usually performed manually, plays a crucial role in validating machine-extracted metadata. However, manual labeling is time-consuming and not always feasible; thus, there is a need to develop automated text labeling techniques in order to accelerate the process of scientific innovation. This need is particularly urgent in fields such as environmental genomics and microbiome science, which have historically received less attention in terms of metadata curation and creation of gold-standard text mining data sets. In this paper, we present two novel automated text labeling approaches for the validation of ML-generated metadata for unlabeled texts, with specific applications in environmental genomics. Our techniques show the potential of two new ways to leverage existing information that is only available for select documents within a corpus to validate ML models, which can then be used to describe the remaining documents in the corpus. The first technique exploits relationships between different types of data sources related to the same research study, such as publications and proposals. The second technique takes advantage of domain-specific controlled vocabularies or ontologies. In this paper, we detail applying these approaches in the context of environmental genomics research for ML-generated metadata validation. Our results show that the proposed label assignment approaches can generate both generic and highly specific text labels for the unlabeled texts, with up to 44% of the labels matching with those suggested by a ML keyword extraction algorithm.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NASA NTRS1995

Machine learning for a Toolkit for Image Mining

A prototype user environment is described that enables a user with very limited computer skills to collaborate with a computer algorithm to develop search tools (agents) that can be used for image analysis, creating metadata for tagging images, searching for images in an image database on the basis of image content, or as a component of computer vision algorithms. Agents are learned in an ongoing, two-way dialogue between the user and the algorithm. The user points to mistakes made in classification. The algorithm, in response, attempts to discover which image attributes are discriminating between objects of interest and clutter. It then builds a candidate agent and applies it to an input image, producing an 'interest' image highlighting features that are consistent with the set of objects and clutter indicated by the user. The dialogue repeats until the user is satisfied. The prototype environment, called the Toolkit for Image Mining (TIM) is currently capable of learning spectral and textural patterns. Learning exhibits rapid convergence to reasonable levels of performance and, when thoroughly trained, Fo appears to be competitive in discrimination accuracy with other classification techniques.

Delanoy, Richard L.↗

DOE OSTI2025

Benchmark Tracking System for Performance Monitoring

Benchmarking is essential for high-performance software development, particularly for monitoring performance across code iterations. This project focused on enhancing the benchmarking process for Lamellar, an asynchronous runtime for High-Performance Computing (HPC) systems developed at Pacific Northwest National Laboratory. Prior to this work, benchmark results were difficult to track and compare across code versions, presenting significant challenges in identifying performance regressions and long-term trends. The primary objective was to establish a systematic, reproducible approach for measuring performance and detecting regressions following code commits. Our methodology involved three key components: standardizing benchmark outputs, implementing data versioning, and developing analysis tools. We standardized the benchmark output format to JSON Line records containing specific fields (execution time, hardware specifications, and environmental variables). To address data management challenges, we evaluated several options and eventually chose a git repository dedicated to benchmark data. We developed a suite of Python tools that processed benchmark results, enriched them with metadata, and facilitated search in the repository. The resulting system enables more efficient filtering and comparison of performance metrics across commit histories, hardware configurations, and benchmark variants through a unified query interface. Our implementation reduces computational overhead by first checking for existing results through configuration matching before initiating new benchmark runs, thereby conserving resources. The system has been validated by Lamellar developers. It organizes results by benchmark type and build configurations for efficient retrieval. Future developments include a planned Large Language Model interface for predicting benchmark performance, incorporating the criterion package for statistical analysis, which will enable automated detection of statistically significant performance changes, and integration with continuous integration pipelines. Despite these enhancements being reserved for future work, this project has successfully provided the Lamellar development team with a framework for maintaining consistent performance standards and identifying optimization opportunities across workloads and hardware environments.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2010

SPASE, Metadata, and the Heliophysics Virtual Observatories

To provide data search and access capability in the field of Heliophysics (the study of the Sun and its effects on the Solar System, especially the Earth) a number of Virtual Observatories (VO) have been established both via direct funding from the U.S. National Aeronautics and Space Administration (NASA) and through other funding agencies in the U.S. and worldwide. At least 15 systems can be labeled as Virtual Observatories in the Heliophysics community, 9 of them funded by NASA. The problem is that different metadata and data search approaches are used by these VO's and a search for data relevant to a particular research question can involve consulting with multiple VO's - needing to learn a different approach for finding and acquiring data for each. The Space Physics Archive Search and Extract (SPASE) project is intended to provide a common data model for Heliophysics data and therefore a common set of metadata for searches of the VO's. The SPASE Data Model has been developed through the common efforts of the Heliophysics Data and Model Consortium (HDMC) representatives over a number of years. We currently have released Version 2.1 of the Data Model. The advantages and disadvantages of the Data Model will be discussed along with the plans for the future. Recent changes requested by new members of the SPASE community indicate some of the directions for further development.

Thieman, James↗

NASA NTRS2006

NASA Taxonomies for Searching Problem Reports and FMEAs

Many types of hazard and risk analyses are used during the life cycle of complex systems, including Failure Modes and Effects Analysis (FMEA), Hazard Analysis, Fault Tree and Event Tree Analysis, Probabilistic Risk Assessment, Reliability Analysis and analysis of Problem Reporting and Corrective Action (PRACA) databases. The success of these methods depends on the availability of input data and the analysts knowledge. Standard nomenclature can increase the reusability of hazard, risk and problem data. When nomenclature in the source texts is not standard, taxonomies with mapping words (sets of rough synonyms) can be combined with semantic search to identify items and tag them with metadata based on a rich standard nomenclature. Semantic search uses word meanings in the context of parsed phrases to find matches. The NASA taxonomies provide the word meanings. Spacecraft taxonomies and ontologies (generalization hierarchies with attributes and relationships, based on terms meanings) are being developed for types of subsystems, functions, entities, hazards and failures. The ontologies are broad and general, covering hardware, software and human systems. Semantic search of Space Station texts was used to validate and extend the taxonomies. The taxonomies have also been used to extract system connectivity (interaction) models and functions from requirements text. Now the Reconciler semantic search tool and the taxonomies are being applied to improve search in the Space Shuttle PRACA database, to discover recurring patterns of failure. Usual methods of string search and keyword search fall short because the entries are terse and have numerous shortcuts (irregular abbreviations, nonstandard acronyms, cryptic codes) and modifier words cannot be used in sentence context to refine the search. The limited and fixed FMEA categories associated with the entries do not make the fine distinctions needed in the search. The approach assigns PRACA report titles to problem classes in the taxonomy. Each ontology class includes mapping words - near-synonyms naming different manifestations of that problem class. The mapping words for Problems, Entities and Functions are converted to a canonical form plus any of a small set of modifier words (e.g. non-uniformity NOT + UNIFORM.) The report titles are parsed as sentences if possible, or treated as a flat sequence of word tokens if parsing fails. When canonical forms in the title match mapping words, the PRACA entry is associated with the corresponding Problem, Entity or Function in the ontology. The user can search for types of failures associated with types of equipment, clustering by type of problem (e.g., all bearings found with problems of being uneven: rough, irregular, gritty ). The results could also be used for tagging PRACA report entries with rich metadata. This approach could also be applied to searching and tagging failure modes, failure effects and mitigations in FMEAs. In the pilot work, parsing 52K+ truncated titles (the test cases that were available), has resulted in identification of both a type of equipment and type of problem in about 75% of the cases. The results are displayed in a manner analogous to Google search results. The effort has also led to the enrichment of the taxonomy, adding some new categories and many new mapping words. Further work would make enhancements that have been identified for improving the clustering and further reducing the false alarm rate. (In searching for recurring problems, good clustering is more important than reducing false alarms). Searching complete PRACA reports should lead to immediate improvement.

Malin, Jane T.↗

NASA NTRSDate not supplied

Application of a Dataset-Publication Knowledge Graph for Improving Earth Science Data Search

Finding a dataset at a NASA data center that is the best fit for the researcher’s application presents a challenge, not only for a novice user but for an experienced one, due to the data complexity and a multitude of choices of the existing data. Users often search for the data based on the application they are interested in, their research domain, phenomena, research topic, etc. As existing dataset metadata may not cover these search terms, the user may not obtain the most relevant results for their purpose. This problem was addressed by leveraging the content of the titles and abstracts of the research papers that utilize NASA datasets. For this, features from the paper titles and abstracts were extracted, and then a knowledge graph (KG) was used to link these features to the datasets used in that paper. The search for the datasets was tested by querying this knowledge graph through various terms extracted from Earth Science ontologies such as Semantic Web for Earth and Environment Technology (SWEET), and it was shown that this KG search outperforms the existing search that exclusively queries the dataset metadata.

Kristina Stoyanova↗

NASA NTRS2018

End-to-End Solution for Data Customization with NASA's Earthdata Search

The goal of NASA's Earthdata Search End-to-End Services workflow is to take the pain and headache out of searching for data and getting that data back in a format that is usable with only that data that is relevant for you. For too long scientists have had to jump through endless hoops, use tools that only offer specific data or specific services, and perform any number of other non-science tasks just to get started on their actual project. Earthdata Search leverages the Common Metadata Repository's (CMR) newly implemented Unified Metadata Models for Services and Variables as well as a new service broker to expose and seamlessly integrate a collection's service capabilities and variables into an intuitive user interface. Using the new End-to-End Services workflow, scientists will be able to quickly see what data is available to be customized, what customization options are available, and actually perform those customizations on the data all within Earthdata Search, regardless of who the data provider is. This talk will demonstrate the simple workflow that will be available to end users and also give an overview covering how the workflow is enabled by the metadata stored within the CMR. (https://search.earthdata.nasa.gov/)

Reese, Mark↗

NASA NTRS2016

Heuristics for Relevancy Ranking of Earth Dataset Search Results

As the Variety of Earth science datasets increases, science researchers find it more challenging to discover and select the datasets that best fit their needs. The most common way of search providers to address this problem is to rank the datasets returned for a query by their likely relevance to the user. Large web page search engines typically use text matching supplemented with reverse link counts, semantic annotations and user intent modeling. However, this produces uneven results when applied to dataset metadata records simply externalized as a web page. Fortunately, data and search provides have decades of experience in serving data user communities, allowing them to form heuristics that leverage the structure in the metadata together with knowledge about the user community. Some of these heuristics include specific ways of matching the user input to the essential measurements in the dataset and determining overlaps of time range and spatial areas. Heuristics based on the novelty of the datasets can prioritize later, better versions of data over similar predecessors. And knowledge of how different user types and communities use data can be brought to bear in cases where characteristics of the user (discipline, expertise) or their intent (applications, research) can be divined. The Earth Observing System Data and Information System has begun implementing some of these heuristics in the relevancy algorithm of its Common Metadata Repository search engine.

science data management↗

NASA NTRS2017

Relevancy Ranking of Satellite Dataset Search Results

science data management↗

NASA NTRSDate not supplied

TOLNet’s FAIR Journey: Yesterday, Today, and Tomorrow

The Tropospheric Ozone Lidar Network (TOLNet) has generated over a decade of ozone vertical profile data products over North America and contributed to several air quality focused field studies. The science value of the TOLNet data has been demonstrated in numerous peer-reviewed publications on air quality and ozone relevant research. As the broad scientific community has moved towards adopting FAIR Principles to make data more findable, accessible, interoperable, and (re)usable, the TOLNet team has been consistently making data more FAIR. This effort has many challenges, partially reflecting on the FAIR principles being domain agnostic while the implementation needs to be domain specific. The FAIR principles declare the dependence on the community standards, domain-relevant metadata, and rich metadata. This presentation uses the TOLNet data and data system as an example to explore the best practices to implement FAIR principle. Particularly, we will examine the metadata and the “richness” to support findability and usability as well as machine-to-machine actionability via API. Last year, as part of our FAIR journey, we launched the TOLNet website (https://tolnet.larc.nasa.gov/) and the API (https://tolnet.larc.nasa.gov/api/). Part of this process included extracting and cataloging metadata across the entire TOLNet mission timeframe. This enabled users to search through the mission by various metadata criteria, improving the findability and accessibility. And computers could connect directly to the TOLNet API to extract both metadata and data, providing a level of interoperability never present before for TOLNet data. On top of that, all new TOLNet data is now automatically validated using the API to ensure it complies with GEOMS standards, aiding in reusability. It takes both technology and scientists working together to make progress. The next step is to evaluate the current TOLNet offerings against NASA’s Practical Guide for Open, Free & FAIR NASA Earth Science Data Products (https://doi.org/10.5067/DOC/ESCO/ESDSWG-0002V1).

TOLNet↗

DOE OSTI2024

Machine Learning (ML) Classifier to Assist Metadata Creation

The Atmospheric Radiation Measurement (ARM) Data Center is responsible for the timely collection, archival, and curation of science data products. These products are freely available through an online data repository. Metadata creation is paramount for scientific users to find and access over seven petabytes of atmospheric science data. The hierarchical metadata structure allows users to search for information at both broad and narrow levels. This project aims to leverage 30 years’ worth of manually created metadata to enable machine predictions of broad-term classifications from narrow-term descriptions. These classification predictions would assist metadata coordinators with their term selections. This paper discusses the cleaning and preprocessing of the training data, the pipeline developed to determine the best model for this task, and the creation of an API metadata classifier for ARM measurement metadata. Our results show that the Linear Support Vector Classification (LinearSVC) algorithm, along with the Term Frequency – Inverse Document Frequency (TF-IDF) vectorizer, is well-suited for our multi-class classification task. Lengthier input training data led to better results, and artificial balancing was unnecessary for this particular use case. This predictive classifier enhances efficiency in metadata creation, as well as supports greater consistency and accuracy in metadata tagging.

Collier, Hannah [ORNL] (ORCID:0000000341284292)↗

NASA NTRS2008

HDF-EOS Web Server

A shell script has been written as a means of automatically making HDF-EOS-formatted data sets available via the World Wide Web. ("HDF-EOS" and variants thereof are defined in the first of the two immediately preceding articles.) The shell script chains together some software tools developed by the Data Usability Group at Goddard Space Flight Center to perform the following actions: Extract metadata in Object Definition Language (ODL) from an HDF-EOS file, Convert the metadata from ODL to Extensible Markup Language (XML), Reformat the XML metadata into human-readable Hypertext Markup Language (HTML), Publish the HTML metadata and the original HDF-EOS file to a Web server and an Open-source Project for a Network Data Access Protocol (OPeN-DAP) server computer, and Reformat the XML metadata and submit the resulting file to the EOS Clearinghouse, which is a Web-based metadata clearinghouse that facilitates searching for, and exchange of, Earth-Science data.

Ullman, Richard↗