Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Catalog”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

CO2 pipeline data catalog

This "Data Catalog" is an excel spreadsheet of collected and catalogued spatial and non-spatial public datasets in relevance to and support of CO2 transport (pipelines) infrastructure development and risk modeling. Includes data associated with CO2 sources, sinks, and transportation.

CO2↗

Data Bookkeeping Service 3: A new event data catalog for CMS

The Data Bookkeeping Service (DBS) provides an event data catalog for Monte Carlo and recorded data of the Compact Muon Solenoid (CMS) Experiment at the Large Hadron Collider (LHC) at CERN, Geneva. It contains all the necessary information used for tracking datasets, like their processing history and associations between runs, files and datasets, on a large scale of about 10(5) datasets and more than 10(7) files. The DBS is widely used within CMS, since all kind of data-processing like Monte Carlo production, processing of recorded event data as well as physics analysis done by the user, are relying on the information stored in DBS.

Giffels, Manuel↗

Data catalog project—A browsable, searchable, metadata system

Modern experiments are typically conducted by large, extended groups, where researchers rely on other team members to produce much of the data they use. The experiments record very large numbers of measurements that can be difficult for users to find, access and understand. Here. we are developing a system for users to annotate their data products with structured metadata, providing data consumers with a discoverable, browsable data index. Machine understandable metadata captures the underlying semantics of the recorded data, which can then be consumed by both programs, and interactively by users. Collaborators can use these metadata to select and understand recorded measurements. The data catalog project is a data dictionary and index which enables users to record general descriptive metadata, use cases and rendering information as well as providing them a transparent data access mechanism (URI). Users describe their diagnostic including references, text descriptions, units, labels, example data instances, author contact information and data access URIs. The list of possible attribute labels is extensible, but limiting the vocabulary of names increases the utility of the system. The data catalog is focused on the data products and complements process-based systems like the Metadata Ontology Provenance project [Greenwald, 2012; Schissel, 2015]. This system can be coupled with MDSplus to provide a simple platform for data driven display and analysis programs. Sites which use MDSplus can describe tree branches, and if desired create ‘processed data trees’ with homogeneous node structures for measurements. Sites not currently using MDSplus can either use the database to reference local data stores, or construct an MDSplus tree whose leaves reference the local data store. A data catalog system can provide a useful roadmap of data acquired from experiments or simulations making it easier for researchers to find and access important data and understand the meaning of the data and how it was obtained. This is particularly useful in research facilities that study the results of many different experiments or simulations and may not know the intricacies of the data organization in use where the data was generated. It is also possible to store a local copy of key data items in local MDSplus trees and then add processed data to the local catalog.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DEEPEN Data Catalog for Magmatic Geothermal Systems in the United States

This data catalog contains information related to the Training Site Analysis for the Geothermica project "DE-risking Exploration of geothermal Plays in magmatic ENvironments (DEEPEN)." The DEEPEN project aims to reduce exploration risk for geothermal fluids in magmatic systems by developing improved an improved framework for interpretation of exploration data using the Play Fairway Analysis (PFA) methodology. The Training Site Analysis performed for DEEPEN leverages existing datasets to develop a customized PFA approach to exploration for multiple geothermal resource types in magmatic systems (conventional hydrothermal resources, supercritical fluid and superheated steam resources, and superhot EGS resources). This data catalog contains links to publicly available data files related to 8 training sites in the United States. US training sites are: the Cascades/Aleutians PFA project; the Hawaii PFA project, the Oregon Cascades PFA project, the Snake River Plain, Idaho PFA project, the Washington State PFA project, Newberry Volcano, Coso Geothermal Field, and the Geysers Geothermal field. This database contains an overview of these training sites, data sources, and links to publicly available exploration datasets. For the five PFA projects, details on exploration data related to PFA components (heat, fluid, permeability, sometimes seal) are provided, including a summary of data weighting methodologies.

15 GEOTHERMAL ENERGY↗

Evaluation of Data Catalog Software for Hanford Site Environmental Datasets

Environmental information and data underpin achievement of the U.S. Department of Energy (DOE) Office of Environmental Management (EM) mission at the Hanford Site. The Hanford Environmental Data Management (HEDM) Program is the DOE Richland Operations Office (RL) approach to develop and implement a formal program for managing environmental data and the associated records, materials, and systems at the Hanford Site. The current project, contract, organization, and contractor-specific efforts at managing environmental data sets are insufficient to provide orderly, long-term, site-wide access. A vital element to be created within the HEDM program plan is a catalog of data sources, called the Hanford Environmental Information and Data Index (HEIDI), that will enable long-term access and retrievability for the multiple independent sources of data that might otherwise be difficult to discover. This report compares leading open source and commercial data catalog platforms using criteria to assess the functionality needed to develop the HEIDI catalog of Hanford data sources that connects and exchanges data with established Hanford Local Area Network (HLAN) enterprise information technology systems. Proprietary platforms evaluated included ArcGIS Enterprise Sites, Junar, OpenDataSoft, and Socrata, and non-proprietary platforms included Energy Data eXchange (EDX), Comprehensive Knowledge Archive Network (CKAN), and DKAN (a Drupal-based open data portal based on CKAN). Capabilities supporting data discoverability, retrieval, and archival, as well as metadata standard requirements and integration into the HLAN were rated as either failing to meet requirements (F), meeting requirements (M), or exceeding requirements by delivering additional desired features (E). The lowest rating for any capability area was assigned as the overall rating for the platform. These findings enable DOE-RL and the contractors implementing the HEDM plan to focus on candidate tools likely to meet the requirements for implementing HEIDI. All of the platforms receiving an overall rating of ‘F’ were unable to be deployed on Hanford infrastructure or within dedicated cloud resources. A propriety software-as-a-service (SaaS) model of delivering a data catalog (e.g., found in software such as Junar and OpenDataSoft) favors consistency across customers at the expense of customization and configurable roles that are needed for Hanford work. Hosting data on a shared commercial platform places limits on dataset size (maximum of 240 Mb for OpenDataSoft), a significant limitation for HEIDI implementation. EDX, a government data catalog based on CKAN, received the ‘F’ rating due to an inability to incorporate authentication from HLAN into the system. Among platforms rated ‘M’ or ‘E’, only the Socrata platform had a SaaS delivery model. In contrast to other SaaS platforms, Socrata provided custom roles and gateways that allow local datasets to be incorporated into an online catalog. Socrata also complies with the Federal Risk and Authorization Management Program, a significant benefit for cloud-based management of Hanford data. The other platforms rated ‘M’ or ‘E’, ArcGIS Enterprise Sites, CKAN, and DKAN, provide fully self-hosted options, allowing for greater control and flexibility with the HEIDI catalog. These widely used tools have supportive communities of practice, extensive customization options, and demonstrated deployments that provide evidence that they can meet requirements, often deliver additional desired features, and work well with federal government systems. Completely customized alternatives built on a collection of applications were not evaluated because achieving similar performance to CKAN or DKAN requires substantial resources, especially in the absence of the active communities that have grown to support these tools. ArcGIS Enterprise Sites, Socrata, CKAN, and DKAN were evaluated as strong candidates for successful implementation with HEIDI.

54 ENVIRONMENTAL SCIENCES↗

A Prototype Software to Demonstrate a Data Catalog for Hanford Environmental Datasets

Ensuring that data on long-term environmental remediation at the Hanford Site is high-quality, traceable, and easily accessible is an ongoing challenge, complicated by decades of data collection, multiple contractors maintaining data sources, and the wide range of data types. A centralized data catalog, known as the Hanford Environmental Information and Data Index (HEIDI), has been under development as part of the Hanford Environmental Data Management (HEDM) program to address these challenges. HEIDI fulfills a critical need to bring together a wide range of data types and sizes from multiple authoritative data sources, while documenting the data pedigree and quality information (i.e., traceable to the data source/originator). This document describes additional development and maturation of the HEIDI prototype. Key accomplishments included deploying the catalog software, Esri Geoportal Server, on a server accessible to Hanford Local Area Network users, conducting cybersecurity evaluations, investigating integrated authentication solutions, and conducting functional testing of the catalog prototype. The server-based deployment enabled targeted feedback, leading to enhancements including improved accessibility features and an expanded metadata schema. Specifications for the server-based deployment of the prototype catalog and the HEIDI metadata schema are provided in this document to support subsequent HEIDI deployment by the U.S. Department of Energy Richland Operations Office.

54 ENVIRONMENTAL SCIENCES↗

Deliverable D12 – Distributed Wind Data Catalog Development Guide and Instruction Manual

Pacific Northwest National Laboratory and Technical University of Denmark completed this deliverable as part of Work Package 2: Data Information Catalog for Distributed Wind Research (WP2) for the International Energy Agency Wind Technology Collaboration Programme Task 41: Enabling Wind to Contribute to a Distributed Energy Future (IEA Wind Task 41). As the final deliverable for WP2, Deliverable D12 includes a data instruction guide for the IEA Wind Task 41 distributed wind data catalog. As such, this document includes: a step-by-step explanation of how the IEA Wind Task 41 data catalog was created, how it was populated, and how to use it; future options for the IEA Wind Task 41 data catalog, and a summary with recommendations for future work.

17 WIND ENERGY↗

TGCat : THE CHANDRA TRANSMISSION GRATING DATA CATALOG AND ARCHIVE

The Chandra Transmission Grating Data Archive and Catalog (TGCat) provides easy access to analysis-ready products, specifically, high-resolution X-ray count spectra and their corresponding calibrations. The web interface makes it easy to find observations of a particular object, type of object, or type of observation; to quickly assess the quality and potential usefulness of the spectra from pre-computed summary plots; or to customize a view with an interactive plotter, optionally combining spectra over multiple orders or observations. Data and responses can be downloaded as a package or as individual files, and the query results themselves can be retrieved as ASCII or Virtual Observatory tables. Portable reprocessing scripts used to create the archive and which use the Chandra X-ray Center's (CXC's) software and other publicly available software are also available, facilitating standard or customized reprocessing from Level 1 CXC archival data to spectra and responses with minimal user interaction.

79 ASTRONOMY AND ASTROPHYSICS↗

HPDF Data Catalog and Lakehouse Demo Heuristic Evaluation and Roadshow Feedback Reports

The December 2025 roadshow gathered rapid feedback from community members on the proof of concept AmSC/HPDF data lakehouse and catalog solution 1. This provided the opportunity to demonstrate our current state of progress and gather input on workflows and AI agents. We recognize that the proofs of concept interfaces (OpenMetadata and Goose) are not intended for our end users so we have captured feedback to keep in mind as additional technical and conceptual design work is undertaken.

97 MATHEMATICS AND COMPUTING↗

Persistence Control of Engineered Functions in Complex Soil Microbiomes (PerCon SFA), Secure Biosystems Design Project Data Catalog at PNNL DataHub

The Persistence Control of Engineered Functions in Complex Soil Microbiomes Project (PerCon SFA) at Pacific Northwest National Laboratory (PNNL) is a Genomic Sciences Program Biosystems Design, Science Focus Area research project consortium. Collaborating across highly integrated institutions, PerCon SFA scientists are exploring how environmental niches can be sculpted using the mechanisms of genome reduction and metabolic addiction to drive secure rhizosphere community design for robust biomass cropping in challenging environments. The PerCon SFA DataHub project repository contains publication-relevant digital dataset and metadata DOI packages, enabling exploration and download of integrated experimental dataset catalogs publicly available to a global scientific community. and metadata repository allow for exploring and downloading integrated experimental biodesign omics dataset catalogs, including experimental protocols and/or workflows, raw and/or processed data, as required by the repository, and other relevant supporting materials and/or metadata required for research reproducibility and reporting.

59 BASIC BIOLOGICAL SCIENCES↗

Specifications and a Prototype Software to Demonstrate a Data Catalog for Hanford Datasets

Environmental management activities at the Hanford Site produce extensive data about site conditions, contaminants, and cleanup activities. Managing, archiving, and accessing that data requires a high degree of collaboration among site contractors and a high level of awareness by project managers and staff. The Hanford Site has a range of databases (e.g., Hanford Environmental Information System [HEIS]) and their associated user interfaces (e.g., Environmental Dashboard Application [EDA], Virtual Library [VL]), as well as other document management systems (e.g., Integrated Document Management System [IDMS]). However, Hanford lacks a single unified resource to find data (which itself comes in multiple formats) amongst the multiple disparate systems, not to mention ad hoc data not contained in an official repository/database.

54 ENVIRONMENTAL SCIENCES↗

Data Catalog Project - A Browsable, Searchable, Metadata System

Modern experiments are typically conducted by large, extended, where researchers rely on other team members to produce much of the data they use. The experiments record very large numbers of measurements which can be difficult for users to find, access and understand. We are developing a system for users to annotate their data products with structured metadata, providing data consumers with a discoverable, browsable data index. Machine understandable metadata captures the underlying semantics of the recorded data, which can then be consumed by both programs, and interactively by users. Collaborators can use these metadata to select and understand recorded measurements.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Faint Object Detection in Multi-Epoch Observations via Catalog Data Fusion

Astronomy in the time-domain era faces several new challenges. One of them is the efficient use of observations obtained at multiple epochs. The work presented here addresses faint object detection and describes an incremental strategy for separating real objects from artifacts in ongoing surveys. The idea is to produce low-threshold single-epoch catalogs and to accumulate information across epochs. This is in contrast to more conventional strategies based on co-added or stacked images. We adopt a Bayesian approach, addressing object detection by calculating the marginal likelihoods for hypotheses asserting that there is no object or one object in a small image patch containing at most one cataloged source at each epoch. The object-present hypothesis interprets the sources in a patch at different epochs as arising from a genuine object; the no-object hypothesis interprets candidate sources as spurious, arising from noise peaks. We study the detection probability for constant-flux objects in a Gaussian noise setting, comparing results based on single and stacked exposures to results based on a series of single-epoch catalog summaries. Our procedure amounts to generalized cross-matching: it is the product of a factor accounting for the matching of the estimated fluxes of the candidate sources and a factor accounting for the matching of their estimated directions. We find that probabilistic fusion of multi-epoch catalogs can detect sources with similar sensitivity and selectivity compared to stacking. The probabilistic cross-matching framework underlying our approach plays an important role in maintaining detection sensitivity and points toward generalizations that could accommodate variability and complex object structure.

79 ASTRONOMY AND ASTROPHYSICS↗

MAGNETIC FIELD COMPONENTS ANALYSIS OF THE SCUPOL 850 μm POLARIZATION DATA CATALOG

We present an extensive analysis of the 850 μm polarization maps of the SCUBA Polarimeter Legacy (SCUPOL) Catalogue produced by Matthews et al., focusing exclusively on the molecular clouds and star-forming regions. For the sufficiently sampled regions, we characterize the depolarization properties and the turbulent-to-mean magnetic field ratio of each region. Similar sets of parameters are calculated from two-dimensional synthetic maps of dust-emission polarization produced with three-dimensional magnetohydrodynamics (MHD) numerical simulations scaled to the S106, OMC-2/3, W49, and DR21 molecular cloud polarization maps. For these specific regions, the turbulent MHD regimes retrieved from the simulations, as described by the turbulent Alfvén and Sonic Mach numbers, are consistent within a factor one to two with the values of the same turbulent regimes estimated from the analysis of Zeeman measurements data provided by Crutcher. Constraints on the values of the inclination angle α of the mean magnetic field with respect to the line of sight are also given. The values obtained from the comparison of the simulations with the SCUPOL data are consistent with the estimates made by using two observational methods provided by other authors. Our main conclusion is that simple, ideal, isothermal, and non-self-gravitating MHD simulations are sufficient in order to describe the large-scale observed physical properties of the envelopes of this set of regions.

79 ASTRONOMY AND ASTROPHYSICS↗

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗