Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “community data standard”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Common practices for quantifying methane emissions from plumes detected by remote sensing

This document provides a set of community-accepted practices for quantifying methane emissions based on plumes detected via spectroscopic remote sensing. Its primary goal is to promote consistency in the generation, validation, reporting, and quality assessment of methane emission estimates derived from remote sensing radiances. Developed by subject matter experts with deep experience across all stages of the measurement process, this guidance reflects a critical evaluation of current methodologies and highlights key practices needed to produce reliable, interoperable, and traceable products. The focus is specifically on methane emissions quantified from distinct plumes originating from localized sources, rather than diffuse emissions spread over large regions, which are beyond the scope of this work. This document is intended to serve both data producers and users. For producers, it offers a framework for aligning with field-recognized standards to ensure their outputs meet rigorous quality and transparency criteria. For users, it provides a reference to assess dataset fitness-for-purpose by highlighting essential metadata, assumptions, and methodological choices that underpin emission estimates. By fostering a shared understanding of best practices, this work aims to enhance comparability, confidence, and utility of remotely sensed methane emission products.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Review of top quark mass measurements in CMS

The top quark mass is one of the most intriguing parameters of the standard model (SM). Its value indicates a Yukawa coupling close to unity, and the resulting strong ties to Higgs physics make the top quark mass a crucial ingredient for understanding essential aspects of the electroweak sector of the SM. This review offers the first comprehensive overview of the top quark mass measurements performed by the CMS Collaboration using the data collected at centre-of-mass energies of 7, 8, and 13 TeV. Moreover, a detailed description of the top quark event reconstruction is provided and dedicated studies of the dominant uncertainties in the modelling of the signal processes are discussed. The interpretation of the experimental results on the top quark mass in terms of the SM Lagrangian parameter is challenging and is a focus of an ongoing discussion in the theory community. The CMS Collaboration has performed two main types of top quark mass measurements, addressing this challenge from different perspectives: highly precise ‘direct’ measurements, based on reconstructed top quark decay products and relying exclusively on Monte-Carlo simulations, as well as ‘indirect’ measurements, where the simulations are employed to determine parton-level cross sections that are compared to fixed-order perturbative calculations. Recent mass extractions using Lorentz-boosted top quarks open a new avenue of measurements based on top quark decay products contained in a single particle jet, with promising prospects for accurate theoretical interpretations.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

An automated multiplexed turbidometric and data collection system for measuring growth kinetics of anaerobes dependent on gaseous substrates

Standard methods of monitoring the growth kinetics of anaerobic microorganisms are generally impractical when there is a protracted or indeterminate period of active growth, and when high numbers of samples or replications are required. As part of our studies of the adaptive evolution of a simple anaerobic syntrophic mutualism, requiring the characterization of many isolates and alternative syntrophic pairings, here we developed a multiplexed growth monitoring system using a combination of commercially available electronics and custom designed circuitry and materials. This system automatically monitors up to 64 sealed, and as needed pressurized, culture tubes and reports the growth data in real-time through integration with a customized relational database. The utility of this system was demonstrated by resolving minor differences in growth kinetics associated with the adaptive evolution of a simple microbial community comprised of a sulfate reducing bacterium, Desulfovibrio vulgaris, grown in syntrophic association with Methanococcus maripaludis, a hydrogenotrophic methanogen.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Long-term trends of impacts of global gasoline and diesel emissions on ambient PM 2.5 and O 3 pollution and the related health burden for 2000–2015

Global economic development and urbanization during the past two decades have driven the increases in demand of personal and commercial vehicle fleets, especially in developing countries, which has likely resulted in changes in year-to-year vehicle tailpipe emissions associated with aerosols and trace gases. However, long-term trends of impacts of global gasoline and diesel emissions on air quality and human health are not clear. In this study, we employ the Community Earth System Model in conjunction with the newly developed Community Emissions Data System as anthropogenic emission inventory to quantify the long-term trends of impacts of global gasoline and diesel emissions on ambient air quality and human health for the period of 2000–2015. Global gasoline and diesel emissions contributed to regional increases in annual mean surface PM 2.5 (particulate matter with aerodynamic diameters ≤2.5 μm) concentrations by up to 17.5 and 13.7 µg m –3 , and surface ozone (O 3 ) concentrations by up to 7.1 and 7.2 ppbv, respectively, for 2000–2015. However, we also found substantial declines of surface PM 2.5 and O 3 concentrations over Europe, the US, Canada, and China for the same period, which suggested the co-benefits of air quality and human health from improving gasoline and diesel fuel quality and tightening vehicle emissions standards. Globally, we estimate the mean annual total PM 2.5- and O 3 -induced premature deaths are 139 700–170 700 for gasoline and 205 200–309 300 for diesel, with the corresponding years of life lost of 2.74–3.47 and 4.56–6.52 million years, respectively. Diesel and gasoline emissions create health-effect disparities between the developed and developing countries, which are likely to aggravate afterwards.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

The CosmoVerse White Paper: Addressing observational tensions in cosmology with systematics and fundamental physics.

The standard model of cosmology has provided a good phenomenological description of a wide range of observations both at astrophysical and cosmological scales for several decades. This concordance model is constructed by a universal cosmological constant and supported by a matter sector described by the standard model of particle physics and a cold dark matter contribution, as well as very early-time inflationary physics, and underpinned by gravitation through general relativity. There have always been open questions about the soundness of the foundations of the standard model. However, recent years have shown that there may also be questions from the observational sector with the emergence of differences between certain cosmological probes. In this White Paper, we identify the key objectives that need to be addressed over the coming decade together with the core science projects that aim to meet these challenges. These discordances primarily rest on the divergence in the measurement of core cosmological parameters with varying levels of statistical confidence. These possible statistical tensions may be partially accounted for by systematics in various measurements or cosmological probes but there is also a growing indication of potential new physics beyond the standard model. After reviewing the principal probes used in the measurement of cosmological parameters, as well as potential systematics, we discuss the most promising array of potential new physics that may be observable in upcoming surveys. We also discuss the growing set of novel data analysis approaches that go beyond traditional methods to test physical models. These new methods will become increasingly important in the coming years as the volume of survey data continues to increase, and as the degeneracy between predictions of different physical models grows. There are several perspectives on the divergences between the values of cosmological parameters, such as the model-independent probes in the late Universe and model-dependent measurements in the early Universe, which we cover at length. The White Paper closes with a number of recommendations for the community to focus on for the upcoming decade of observational cosmology, statistical data analysis, and fundamental physics developments.

Dienes, Keith [Univ. of Arizona, Tucson, AZ (Unite↗
DOE OSTI2022

Transit Time Estimation in Catchments: Recent Developments and Future Directions

Abstract Water transit time is now a standard measure in catchment hydrological and ecohydrological research. The last comprehensive review of transit time modeling approaches was published 15+ years ago. But since then the field has largely expanded with new data, theory and applications. Here, we review these new developments with focus on water‐age‐balance approaches and data‐based approaches. We discuss and compare methods including StorAge‐Selection functions, well/partially mixed compartments, water age tracking through spatially distributed models, direct transit time estimates from controlled experiments, young water fractions, and ensemble hydrograph separation. We unify some of the heterogeneity in the literature that has crept in with these many new approaches, in an attempt to clarify the key differences and similarities among them. Finally, we point to open questions in transit time research, including what we still need from theory, models, field work, and community practice.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

How Should Researchers Measure Perovskite-Based Monolithic Multijunction Solar Cells’ Performance? A Calibration Lab's Perspective

Perovskite multijunctions (PVSK MJs) have made remarkable progress with monolithic PVSK/PVSK tandems surpassing the efficiency of single-junction (1 J) PVSK cells and PVSK/Si cells reported to exceed the 30% efficiency mark. These efficiencies are reported at standard test conditions (STC), established by the photovoltaic (PV) community to facilitate comparison between devices and technologies. Herein, it is discussed why an accurate STC performance measurement for a MJ is more complicated than for a 1 J cell and the special aspects to be considered when measuring the current-voltage characteristics and the performance of PVSK-containing MJ cells are emphasized. It is discussed why a spectrally adjustable solar simulator is needed and the sequence of accurate performance measurement, namely measurement of the spectral response of each junction, adjustment of the spectrum, and appropriate protocols for measuring the power output of the device at STC, is presented. For all these, common errors and pitfalls that sometime lead to misleading interpretation of the results are presented, and the methods to evaluate the accuracy of the data when a spectrally adjustable solar simulator is not available are recommended. Finally, first step is taken toward recommending performance measurement approaches when high throughput is required as will eventually be in a production line.

14 SOLAR ENERGY↗
DOE OSTI2025

The Aerosol Model Benchmarking Repository: A toolkit for model intercomparison

The Aerosol Model Benchmarking Repository and Standards (AMBRS) project was initiated to provide tools and to establish community standards for benchmarking aerosol models. This report describes a set of open-source tools for building, running, and analyzing aerosol box model simulations in a standardized framework. The framework consists of three core components: AMBuilder, a CMake-based build system that compiles supported models consistently; AMBRS, a Python module that defines unified numerical experiments and executes them with aligned inputs; and PyParticle, an aerosol analysis package that standardizes output, computes diagnostics, and visualizes simulation results. Together, these tools enable reproducible intercomparison of aerosol schemes and support process-level evaluation of how model simplifications affect predictions of size distributions, cloud condensation nuclei activity, and other relevant properties relevant for the Earth-Energy system. Beyond its role in benchmarking, AMBRS provides a platform for studying aerosol processes across scales and can be used to generate training data for AI/ML applications in support of a broader hierarchical aerosol modeling strategy.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

STREAMS guidelines: standards for technical reporting in environmental and host-associated microbiome studies

The interdisciplinary nature of microbiome research, coupled with the generation of complex multi-omics data, makes knowledge sharing challenging. The Strengthening the Organization and Reporting of Microbiome Studies (STORMS) guidelines provide a checklist for the reporting of study information, experimental design and analytical methods within a scientific manuscript on human microbiome research. Here, in this Consensus Statement, we present the standards for technical reporting in environmental and host-associated microbiome studies (STREAMS) guidelines. The guidelines expand on STORMS and include 67 items to support the reporting and review of environmental (for example, terrestrial, aquatic, atmospheric and engineered), synthetic and non-human host-associated microbiome studies in a standardized and machine-actionable manner. Based on input from 248 researchers spanning 28 countries, we provide detailed guidance, including comparisons with STORMS, and case studies that demonstrate the usage of the STREAMS guidelines. In conclusion, STREAMS, like STORMS, will be a living community resource updated by the Consortium with consensus-building input of the broader community.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Divide and conquer: using RhizoVision Explorer to aggregate data from multiple root scans using image concatenation and statistical methods

Roots are important in agricultural and natural systems for determining plant productivity and soil carbon inputs. Sometimes, the amount of roots in a sample is too much to fit into a single scanned image, so the sample is divided among several scans, and there is no standard method to aggregate the data. Here, we describe and validate two methods for standardizing measurements across multiple scans: image concatenation and statistical aggregation. We developed a Python script that identifies which images belong to the same sample and returns a single, larger concatenated image. These concatenated images and the original images were processed with RhizoVision Explorer, a free and open-source software. An R script was developed, which identifies rows of data belonging to the same sample and applies correct statistical methods to return a single data row for each sample. These two methods were compared using example images from switchgrass, poplar, and various tree and ericaceous shrub species from a northern peatland and the Arctic. Most root measurements were nearly identical between the two methods except median diameter, which cannot be accurately computed by statistical aggregation. We believe the availability of these methods will be useful to the root biology community.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Versailles Project on Advanced Materials and Standards interlaboratory study on intensity calibration for x-ray photoelectron spectroscopy instruments using low-density polyethylene

We report the results of a Versailles Project on Advanced Materials and Standards interlaboratory study on the intensity scale calibration of x-ray photoelectron spectrometers using low-density polyethylene (LDPE) as an alternative material to gold, silver, and copper. An improved set of LDPE reference spectra, corrected for different instrument geometries using a quartz-monochromated Al Kα x-ray source, was developed using data provided by participants in this study. Using these new reference spectra, a transmission function was calculated for each dataset that participants provided. When compared to a similar calibration procedure using the NPL reference spectra for gold, the LDPE intensity calibration method achieves an absolute offset of ~3.0% and a systematic deviation of ±6.5% on average across all participants. For spectra recorded at high pass energies (≥90 eV), values of absolute offset and systematic deviation are ~5.8% and ±5.7%, respectively, whereas for spectra collected at lower pass energies (<90 eV), values of absolute offset and systematic deviation are ~4.9% and ±8.8%, respectively; low pass energy spectra perform worse than the global average, in terms of systematic deviations, due to diminished count rates and signal-to-noise ratio. Differences in absolute offset are attributed to the surface roughness of the LDPE induced by sample preparation. We further assess the usability of LDPE as a secondary reference material and comment on its performance in the presence of issues such as variable dark noise, x-ray warm up times, inaccuracy at low count rates, and underlying spectrometer problems. In response to participant feedback and the results of the study, we provide an updated LDPE intensity calibration protocol to address the issues highlighted in the interlaboratory study. Finally, we also comment on the lack of implementation of a consistent and traceable intensity calibration method across the community of x-ray photoelectron spectroscopy (XPS) users and, therefore, propose a route to achieving this with the assistance of instrument manufacturers, metrology laboratories, and experts leading to an international standard for XPS intensity scale calibration.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Imagery Applications for Advanced Event Analytics (WBS 24.3.1.3.3-IDC FY202 Final Project Report)

Accurate event locations and replicability of location analyses are essential for assessing the nature of an event, its context, ambient site conditions, and proximity to relevant facilities and infrastructure. Additionally, accurate event locations provide valuable information that reduce uncertainties, improve confidence in event analyses, and inform in-field verification activities. However, event location/relocation and replicability are difficult due to a number of factors, including spatially-sparse network coverage in some areas of the globe and variability in seismic data processing. This team proposed that the incorporation of high-fidelity imagery as a data backbone to the analytical assessment of a suspected underground explosion and/or an advanced seismic event bulletin produced by the International Data Centre (IDC) of the Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO PrepCom) could reduce uncertainties and improve confidence in analyses. Specifically, temporally-separated images can reduce uncertainty by identifying areas where change has occurred (e.g., building construction or demolition, road or facilities improvements). The primary goal of this project was to develop an automated geospatial processing script for imagery change detection to better reflect needs of the technical community (including the IDC) and to make the use of such a tool accessible in a variety of settings across platforms. Technical experts at Los Alamos National Laboratory successfully built GAIA: the Geospatial Automated Imagery Analysis tool, to fill this need. GAIA combines five tool components to produce orthorectified time separated imagery and imagery change detection maps. Our toolkit (1) reduces error by providing a standardized workflow for image analyses and (2) significantly reduces processing time from between 7 and 24+ hours to approximately 5 minutes. Technical experts at Sandia National Laboratories supported GAIA via beta-testing and by introducing a web-based system approach for increased applicability. To test the function, performance, broad application, and ease-of-use of GAIA, we applied it to four separate test cases. The results of this preliminary investigation show promise in reducing uncertainty in seismic event locations: if satellite imagery can show regions where operations that produce seismic activity likely occurred, then pursuing imagery to locate epicenters of seismic nuclear events could reduce the time needed to find the true epicenter location.

58 GEOSCIENCES↗
DOE OSTI2026

Evaluating hydrogen permeation through medium-density and high-density polyethylene pipes to enable a hydrogen-compatible infrastructure

As the United States and the global community pursue energy abundance and resilience, hydrogen is becoming a critical energy carrier. The future of hydrogen transport depends on one thing: verifying that today’s pipelines are hydrogen ready. This study investigates the permeation behavior of medium-density polyethylene (MDPE) and high-density polyethylene (HDPE) pipelines under pure hydrogen gas environment at various pressures and temperatures. Using thermal desorption analysis (TDA) and direct pipe permeability measurements, hydrogen loss rates are quantified, revealing minimal permeation losses (< 10.5 g/km/day) under realistic operating conditions. Effects of temperature and pressure with Standard Dimension Ratios (SDR), were evaluated to determine their influence on permeation rates of MDPE and HDPE pipelines. In addition, ex-situ density and degree of crystallinity (DOC) of MDPE and HDPEs after hydrogen exposure were also investigated to understand pressure-dependency of polymer properties. The findings provide essential data for assessing polymer-based pipeline materials for safe and reliable hydrogen transport.

08 HYDROGEN↗
DOE OSTI2020

Imagery Applications for Advanced Event Analytics (WBS 24.3.1.3.3-IDC FY2020 Final Project Report)

Accurate event locations and replicability of location analyses are essential for assessing the nature of an event, its context, ambient site conditions, and proximity to relevant facilities and infrastructure. Additionally, accurate event locations provide valuable information that reduce uncertainties, improve confidence in event analyses, and inform in-field verification activities. However, event location/relocation and replicability are difficult due to a number of factors, including spatially-sparse network coverage in some areas of the globe and variability in seismic data processing. This team proposed that the incorporation of high-fidelity imagery as a data backbone to the analytical assessment of a suspected underground explosion and/or an advanced seismic event bulletin produced by the International Data Centre (IDC) of the Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO PrepCom) could reduce uncertainties and improve confidence in analyses. Specifically, temporally-separated images can reduce uncertainty by identifying areas where change has occurred (e.g., building construction or demolition, road or facilities improvements). The primary goal of this project was to develop an automated geospatial processing script for imagery change detection to better reflect needs of the technical community (including the IDC) and to make the use of such a tool accessible in a variety of settings across platforms. Technical experts at Los Alamos National Laboratory successfully built GAIA: the Geospatial Automated Imagery Analysis tool, to fill this need. GAIA combines five tool components to produce orthorectified time-separated imagery and imagery change detection maps. Our toolkit (1) reduces error by providing a standardized workflow for image analyses and (2) significantly reduces processing time from between 7 and 24+ hours to approximately 5 minutes. Technical experts at Sandia National Laboratories supported GAIA via beta-testing and by introducing a web-based system approach for increased applicability. To test the function, performance, broad application, and ease-of-use of GAIA, we applied it to four separate test cases. The results of this preliminary investigation show promise in reducing uncertainty in seismic event locations: if satellite imagery can show regions where operations that produce seismic activity likely occurred, then pursuing imagery to locate epicenters of seismic nuclear events could reduce the time needed to find the true epicenter location.

58 GEOSCIENCES↗
DOE OSTI2022

Model-Form Epistemic Uncertainty Quantification for Modeling with Differential Equations: Application to Epidemiology

Modeling real-world phenomena to any degree of accuracy is a challenge that the scientific research community has navigated since its foundation. Lack of information and limited computational and observational resources necessitate modeling assumptions which, when invalid, lead to model-form error (MFE). The work reported herein explored a novel method to represent model-form uncertainty (MFU) that combines Bayesian statistics with the emerging field of universal differential equations (UDEs). The fundamental principle behind UDEs is simple: use known equational forms that govern a dynamical system when you have them; then incorporate data-driven approaches – in this case neural networks (NNs) – embedded within the governing equations to learn the interacting terms that were underrepresented. Utilizing epidemiology as our motivating exemplar, this report will highlight the challenges of modeling novel infectious diseases while introducing ways to incorporate NN approximations to MFE. Prior to embarking on a Bayesian calibration, we first explored methods to augment the standard (non-Bayesian) UDE training procedure to account for uncertainty and increase robustness of training. In addition, it is often the case that uncertainty in observations is significant; this may be due to randomness or lack of precision in the measurement process. This uncertainty typically manifests as “noisy” observations which deviate from a true underlying signal. To account for such variability, the NN approximation to MFE is endowed with a probabilistic representation and is updated using available observational data in a Bayesian framework. By representing the MFU explicitly and deploying an embedded, data-driven model, this approach enables an agile, expressive, and interpretable method for representing MFU. In this report we will provide evidence that Bayesian UDEs show promise as a novel framework for any science-based, data-driven MFU representation; while emphasizing that significant advances must be made in the calibration of Bayesian NNs to ensure a robust calibration procedure.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Effects of forest structural and compositional change on forest microclimates across a gradient of disturbance severity

Forest structural diversity and community composition are key in regulating forest microclimates. When disturbance affects structural diversity or composition, forest microclimates may be altered due to changes in soil temperature, soil water content, and light availability. It is unclear however which structural or compositional components, when changed or to what extent, result in microclimatic change. To address this question, we used data from a large scale, manipulative stem-girdling experiment in northern, lower Michigan—the Forest Resilience and Threshold Experiment (FoRTE). FoRTE follows a factorial design with multiple levels of disturbance severity (0, 45, 65, 85%) based on targeted reductions in gross leaf area index via stem-girdling induced mortality. These disturbance severity treatments are applied in two ways: either as top-down (largest trees are killed) or bottom-up (small to medium trees killed) treatments. We examined how multiple components of structural diversity and community composition changed as a product of disturbance severity and type, and then tested for resulting effects on forest microclimates (light availability, soil temperature, and soil water), using a multivariate, Random Forest framework. We found that measures of community composition (species richness, species evenness, and Shannon-Wiener Diversity Index) and stand structure (basal area, standard deviation of DBH, tree size diversity) declined more following disturbance than did measures of canopy cover, heterogeneity, arrangement, or height. However, when changes in each variable from pre- to post-disturbance, measured as log change, were employed in a multivariate, Random Forest regression framework, structural diversity measures of heterogeneity (rugosity, top rugosity), cover (canopy cover), and arrangement (porosity) were the most influential variables, but with differences among bottom-up and top-down treatments We found that the death of large trees from disturbance impacts soil temperature, water, and light environments more substantially and uniformly across disturbance gradients than does the death of smaller trees. Furthermore, our results have implications for both statistical and process-based modeling of forest disturbance.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Automated Integration of Continental-Scale Observations in Near-Real Time for Simulation and Analysis of Biosphere–Atmosphere Interactions

The National Ecological Observatory Network (NEON) is a continental-scale observatory with sites across the US collecting standardized ecological observations that will operate for multiple decades. To maximize the utility of NEON data, we envision edge computing systems that gather, calibrate, aggregate, and ingest measurements in an integrated fashion. Edge systems will employ machine learning methods to cross-calibrate, gap-fill and provision data in near-real time to the NEON Data Portal and to High Performance Computing (HPC) systems, running ensembles of Earth system models (ESMs) that assimilate the data. For the first time gridded EC data products and response functions promise to offset pervasive observational biases through evaluating, benchmarking, optimizing parameters, and training new machine learning parameterizations within ESMs all at the same model-grid scale. Leveraging open-source software for EC data analysis, we are already building software infrastructure for integration of near-real time data streams into the International Land Model Benchmarking (ILAMB) package for use by the wider research community. We will present a perspective on the design and integration of end-to-end infrastructure for data acquisition, edge computing, HPC simulation, analysis, and validation, where Artificial Intelligence (AI) approaches are used throughout the distributed workflow to improve accuracy and computational performance.

Durden, David J.↗
DOE OSTI2025

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗