sandialabs/xyz
Tools for extracting and viewing information from zip archives. Simply extracts and displays zip file metadata in a format convenient for analysts.SAND2020-0168 M
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Tools for extracting and viewing information from zip archives. Simply extracts and displays zip file metadata in a format convenient for analysts.SAND2020-0168 M
Laika BOSS (Binary Object Scanning System), often written as "Laikaboss", is an object scanner and intrusion detection system that strives to be scalable, flexible, and verbose. Its goal is to take arbitrary file input and extract all relevant subfiles and metadata for use in helping analysts determine the function and maliciousness of a file at a glance. This is an extension to the original code developed by Lockheed Martin.
Statement of purpose: Cyclones alter the function and composition of tropical forests, making effects of intensifying cyclones on carbon-rich forests a critical topic of study. Here, we quantified cyclone-induced damage and recovery of 21 cyclone disturbances affecting 23 pantropical forest sites between 1988-2017 utilizing leaf area index (LAI), enhanced vegetation index (EVI), normalized difference vegetation index (NDVI), and transformed NDVI (kNDVI) values from Google Earth Engine. Field observations collected in a meta-analysis (Bomfim et al., 2022, in review) were used to ground-truth and test effects of soil resource availability and disturbance factors on damage and recovery. This meta-analysis also served as the basis to begin vegetation index extraction, utilizing unique site and date combinations, from tropical forests effect by cyclone disturbances. We began collecting NDVI (5km resolution) from the NOAA Climate Data Record (CDR) of AVHRR Normalized Difference Vegetation Index (NDVI), Version 5 data product (Vermote, 2019) for all case studies included, 42. Next, we began extracting Landsat data from Landsat 4, 5, and 8, courtesy of the U.S. Geological Survey, in search of higher resolution data. We selected a 3 by 3 Landsat pixel area, leading to a 90m resolution data extraction. The specific imagery used includes Landsat 4 USGS Landsat 4 TM Collection 1 Tier 1 TOA (top of atmosphere) Reflectance, Landsat 5 USGS Landsat 5 TM (thematic mapper) Collection 1 Tier 1 TOA Reflectance, and Landsat 8 USGS Landsat 8 Collection 1 Tier 1 TOA Reflectance. Within Google Earth Engine, we selected the date and location (latitude and longitude), calculated NDVI, kNDVI, and EVI utilizing Landsat bands (see metadata_NGEE-tropics_cyclones), and extracted post- and pre-cyclone values for each case study to calculate cyclone-induced change in the vegetative index. Due to limited spatial resolution of Landsat remote sensing data, MODIS products were investigated next. First, the MOD13Q1.006 Terra Vegetation Indices 16-Day Global 250m product was used to extract 250m EVI and NDVI (Didan, 2015) and then the MCD15A3H.006 MODIS Leaf Area Index/FPAR 4-Day Global 500m product product was used to extract LAI 500m (Myneni et al., 2015). Pre- and post-cyclone values, change in the vegetative index, and standard deviation for all values are included in the main csv (see case_study_data.csv) for all vegetative indices collected, including LAI 500m, EVI 250m, NDVI 250m, NDVI 90m, kNDVI 90m, EVI 90m, and NDVI 5km. Lastly, recovery values were calculated utilizing a standardization method (see metadata_NGEE-tropics_cyclones) and recovery values for MODIS (see MODIS_recovery.csv) and Landsat (Landsat_recovery.csv) data are included.
SAND2026-16981O Bibcheck is designed to extract bibliographies from research papers and perform metadata searches to identify errors. It assists authors in checking their bibliographies for metadata errors during the writing process and helps reviewers identify errors in bibliographies of papers under review. The software uses large language models (LLMs) to extract bibliography entries from PDF documents, classifies the type of bibliography entry, and verifies referenced works. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
This data set contains measurements of soil characteristics (aggregate size distribution and mean size, total aggregate associated carbon, extractable organic C, and microbial biomass C), microbial respiration, and soil metabolite concentrations from a transient and steady soil moisture incubation experiment using soils of different textures (sandy, loamy, and clayey). The study investigated mechanisms driving the Birch effect (increased carbon mineralization pulses with wetting following a drying period) in differing soil textures. Three different soils of distinctly different textures were collected from 0-15cm depth in Georgia (sandy, 2017-05-01), Missouri (loamy, 2017-06-14), and Texas (clayey, December 2017). Soils were incubated for 140 days with destructive harvests done on days 1, 29, 33, 56, 112, 116, and 140 in transient soil moisture incubation and on days 1, 33, 116, and 140 in steady state soil moisture incubation. This dataset contains six data files in comma separate (.csv) format. Additional metadata are provided: six data dictionaries and a file-level metadata file in comma separate (*.csv) format and a user guide in PDF (*.pdf) format.
This data contains data from 90-day long incubation study which aimed to look at the soil moisture-texture relationship on soil organic carbon (SOC) cycling. Soils were collected from three distinct soil textures from mixed forests in 2017: sandy (Georgia, 2017-05-01), loamy (Missouri, 2017-06-14) and clayey (Texas, December 2017) were incubated at different soil moisture levels (air-dried, 25% water holding capacity (WHC), 50% WHC, 100% WHC and 175% WHC) at room temperature for a period of 90 days. Files contain microbial respiration, active and slow SOC pools, and their respective mineralization rates, extractable organic carbon (C), and C-acquiring extracellular enzymes. Findings from these data were used in Singh et al. (2021). This study aimed to examine the interactive effect of soil moisture and texture on SOC mineralization. Soil samples of three distinct textures (sandy, loamy, and clayey) were collected from mixed forests of Georgia, Missouri, and Texas, respectively. Soil cores of 5 cm diameter were collected from numerous random locations at each site from 0-15 cm depth after scraping the litter layer and mixed thoroughly to obtain a composite sample per site. Three additional soil cores were collected to determine the WHC using pressure plate extractors. Soil samples were composited, and triplicate soil samples were incubated in mason jars for a period of 90 days at room temperature under different moisture regimes: air dried, 25% WHC, 50% WHC, at WHC and 100% saturation. Soil respiration was measured weekly, and destructive sampling was conducted at 1, 15, 60, and 90 days to determine extractable organic C, C acquiring enzyme activity, and active and slow SOC pools with their respective mineralization rates. The C acquiring enzyme activity was the total activity of α-glucosidase, β-glucosidase, cellobiohydrolase, and β-xylosidase enzymes. Gas samples for microbial respiration measurements were collected from headspace of incubation jars through the sampling ports on the lids and then analyzed using a Shimadzu Gas Chromatograph (GC-2014). Prior to sampling, the vials were evacuated. Blank correction was also done by collecting gas samples from empty incubation jars. Double pool exponential decay model was used in SigmaPlot to determine the active and slow SOC pools and their mineralization rates (Farrar et al., 2012; Jagadamma et al., 2014). The C-acquiring extracellular enzymes were measured using the microplate method by German et al., (2011). Microbial community structure was determined using the phospholipid fatty acid (PLFA) and neutral lipid fatty acid (NLFA) analyses (Buyer and Sasser, 2012). This dataset has seven data files provided in comma-separate (*.csv) format. Additional metadata are provided: seven data dictionaries and a file-level metadata file in comma separate (*.csv) format and a user guide in PDF (*.pdf) format.
Abstract In an era of rapid advancement of algorithms that extract knowledge from data, data and metadata management are increasingly critical to research success. In materials science, there are few examples of experimental databases that contain many different types of information, and compared with other disciplines, the database sizes are relatively small. Underlying these issues are the challenges in managing and linking data across disparate synthesis and characterization experiments, which we address with the development of a lightweight data management framework that is generally applicable for experimental science and beyond. Five years of managing experiments with this system has yielded the Materials Experiment and Analysis Database (MEAD) that contains raw data and metadata from millions of materials synthesis and characterization experiments, as well as the analysis and distillation of that data into property and performance metrics via software in an accompanying open source repository. The unprecedented quantity and diversity of experimental data are searchable by experiment and analysis attributes generated by both researchers and data processing software. The search web interface allows users to visualize their search results and download zipped packages of data with full annotations of their lineage. The enormity of the data provides substantial challenges and opportunities for incorporating data science in the physical sciences, and MEAD’s data and algorithm management framework will foster increased incorporation of automation and autonomous discovery in materials and chemistry research.
We will describe a component of the Intelligent Data Delivery Service being developed in collaboration with IRIS-HEP and the LHC experiments. ServiceX is an experiment-agnostic service to enable on-demand data delivery specifically tailored for nearly-interactive vectorized analysis. This work is motivated by the data engineering challenges posed by HL-LHC data volumes and the increasing popularity of python and Spark-based analysis workflows.ServiceX gives analyzers the ability to query events by dataset metadata. It uses containerized transformations to extract just the data required for the analysis. This operation is colocated with the data to avoid transferring unnecessary branches over the WAN. Simple filtering operations are supported to further reduce the amount of data transferred.Transformed events are cached in a columnar datastore to accelerate delivery of subsequent similar requests. ServiceX will learn commonly related columns and automatically include them in the transformation to increase the potential for cache hits by other users.Selected events are streamed to the analysis system using an efficient wire protocol that can be readily consumed by a variety of computational frameworks. This reduces time-to-insight for physics analysis by delegating to ServiceX the complexity of event selection, slimming, reformatting, and streaming.
The management of large geophysical and celestial data bases is now, more than ever, the most critical path to timely data analysis. With today's large volume data sets from multiple satellite missions, analysts face the task of defining useful data bases from which data and metadata (information about data) can be extracted readily in a meaningful way. Visualization, following an object-oriented design, is a fundamental method of organizing and handling data. Humans, by nature, easily accept pictorial representations of data. Therefore graphically oriented user interfaces are appealing, as long as they remain simple to produce and use. The Visual Interface for Space and Terrestrial Analysis (VISTA) system, currently under development at the Naval Research Laboratory's Backgrounds Data Center (BDC), has been designed with these goals in mind. Its graphical user interface (GUI) allows the user to perform queries, visualization, and analysis of atmospheric and celestial backgrounds data.
This paper will be submitted to the proceedings The Eleventh IASTED International Conference on. Signal and Image Processing. Given a video or video archive how does one effectively and quickly summarize, classify, and search the information contained within the data? This paper addresses these issues by describing a process for the automated generation of a table-of-contents and keyword, topic-based index tables that can be used to catalogue, summarize, and search large amounts of video data. Having the ability to index and search the information contained within the videos, beyond just metadata tags, provides a mechanism to extract and identify "useful" content from image and video data.
This software can track multiple moving objects within a video stream simultaneously, use visual features to aid in the tracking, and initiate tracks based on object detection in a subregion. A simple programmatic interface allows plugging into larger image chain modeling suites. It extracts unique visual features for aid in tracking and later analysis, and includes sub-functionality for extracting visual features about an object identified within an image frame. Tracker Toolkit utilizes a feature extraction algorithm to tag each object with metadata features about its size, shape, color, and movement. Its functionality is independent of the scale of objects within a scene. The only assumption made on the tracked objects is that they move. There are no constraints on size within the scene, shape, or type of movement. The Tracker Toolkit is also capable of following an arbitrary number of objects in the same scene, identifying and propagating the track of each object from frame to frame. Target objects may be specified for tracking beforehand, or may be dynamically discovered within a tripwire region. Initialization of the Tracker Toolkit algorithm includes two steps: Initializing the data structures for tracked target objects, including targets preselected for tracking; and initializing the tripwire region. If no tripwire region is desired, this step is skipped. The tripwire region is an area within the frames that is always checked for new objects, and all new objects discovered within the region will be tracked until lost (by leaving the frame, stopping, or blending in to the background).
Forest allocation of net primary productivity (NPP) to reproduction is poorly quantified globally, despite its critical role in forest regeneration and a well-supported trade-off with allocation to growth. Although field measurements of total NPP are rare, our work finds that a proxy for reproductive carbon allocation constructed from leaf (L) and reproductive (R) litterfall fluxes, R/(R+L), is strongly correlated with R/NPP, facilitating analysis across a wide range of sites where biometric estimates of NPP are not available (R² = 0.85; Hanbury-Brown et al., 2022, Ward et al., in prep). To investigate relationships between ecosystem-scale reproductive allocation (RA) and climate, soil fertility, and stand age gradients, we conducted a literature search and synthesized 824 observations of annual average leaf and reproductive litterfall fluxes across forest sites globally. The zip file includes 1) a folder Data/ containing the litterfall data ("GlobalForestRA_data.csv") and metadata ("GlobalForestRA_metadata.doc") files. The data file includes geographic coordinates, long-term mean annual temperature and precipitation (1970-2000, extracted from WorldClim2.1), leaf and reproductive litterfall fluxes, sampling interval and protocols, forest characteristics (dominant leaf morphology, information pertaining to forest age and successional stage, and disturbance history) and soil properties (% sand, %silt, %clay, total phosphorus (P), nitrogen (N), cation exchange capacity (CEC) and pH) extracted from SoilGrids250 and from on-site measurements, where available. The metadata file contains information about each variable reported in the data file, including data sources, processing methods, and all references. The Data folder contains two additional files used to create Figure 1; these are described in greater detail in the README.2) R scripts GloalForestRA_analysis.r and GlobalForestRA_SI.r and a folder /Functions used to produce results, figures, and tables in the manuscript Ward et al. (in press)3) a README file describing how the data and R scripts can be used to reproduce statistical results, figures, and tables found in the manuscript. Ward et al. (in press)This repository can also be found at: https://github.com/r-ward/Global_Analysis_ForestRA.Ward, R.E., Zhang-Zheng, H. Aernethy, K., Adu-Bredu, S., Arroyo, L., Bailey, A. et al. (in press). Forest age rivals climate to explain reproductive allocation patterns in forest ecosystems globally. Ecology Letters. Hanbury-Brown, A.R., Ward, R.E. & Kueppers, L.M. (2022). Forest regeneration within Earth system models: current process representations and ways forward. New Phytol., 235, 20–40.Ward et al. (2025), Forest age rivals climate to explain reproductive allocation patterns in forest ecosystems globally, in prep.
During the first LHC run, CMS stored about one hundred petabytes of data. Storage accounting and monitoring help to meet the challenges of storage management, such as efficient space utilization, fair share between users and groups and resource planning. We present a newly developed CMS space monitoring system based on the storage metadata dumps produced at the sites. The information extracted from the storage dumps is aggregated and uploaded to a central database. A web based data service is provided to retrieve the information for a given time interval and a range of sites, so it can be further aggregated and presented in the desired format. The system has been designed based on the analysis of CMS monitoring requirements and experiences of the other LHC experiments. In this paper, we demonstrate how the existing software components of the CMS data placement system, PhEDEx, have been re-used, dramatically reducing the development effort.
The North Railroad Avenue Plume (NRAP) Superfund site in New Mexico, USA exemplifies successful chlorinated solvent bioremediation. NRAP was the result of leakage from a dry-cleaning that operated for 37 years. The presence of tetrachloroethene biodegradation byproducts, organohalide respiring genera (OHRG), and reductive dehalogenase (rdh) genes detected in groundwater samples indicated that enhanced reductive dechlorination (ERD) was the remedy of choice. This was achieved through biostimulation by mixing emulsified vegetable oil into the contaminated aquifer. This report combines metagenomic techniques with site monitoring metadata to reveal new details of ERD. DNA extracts from groundwater samples collected prior to and at four, 23 and 39 months after remedy implementation were subjected to whole metagenome sequencing (WMS) and 16S rRNA gene amplicon (16S) analyses. The response of the indigenous NRAP microbiome to ERD protocols is consistent with results obtained from microcosms, dechlorinating consortia, and observations at other contaminated sites. WMS detects three times as many phyla and six times as many genera as 16S. Both techniques reveal abundance changes in Dehalococcoides and Dehalobacter that reflect organohalide form and availability. Methane was not detected before biostimulation but appeared afterwards, corresponding to an increase in methanogenic Archaea. Assembly of WMS reads produced scaffolds containing rdh genes from Dehalococcoides, Dehalobacter, Dehalogenimonas, Desulfocarbo, and Desulfobacula. Anaerobic and aerobic cometabolic organohalide degrading microbes that increase in abundance include methanogenic Archaea, methanotrophs, Dechloromonas, and Xanthobacter, some of which contain hydrolytic dehalogenase genes. Aerobic cometabolism may be supported by oxygen gradients existing in aquifer microenvironments or by microbes that produce O 2 via microbial dismutation. The NRAP model for successful ERD is consistent with the established pathway and identifies new taxa and processes that support this syntrophic process. This project explores the potential of metagenomic tools (MGT) as the next advancement in bioremediation.
We are collecting long-term leaf phenology data at The Morton Arboretum to determine seasonal patterns of leaf production in trees. This data on leaf phenology will be integrated with other ongoing data streams to create a connection between above- and below-ground tree processes. This data package contains raw and smooth outputs from phenology data, as well as extracted phenophase dates (i.e., start, peak, and end of season): the raw and smooth outputs from the PhenoCam GUI can be found in the "leafRaw.csv" and "leafSmooth.csv" files, respectively, and the extracted phenophase dates can be found in the "leafPhenophaseDates2019-2023.csv" file. Extracted phenophase dates for evergreen species in 2023 are currently unavailable, and the files will be updated once they are extracted. Additional information on units and other file-level metadata can be found within each data file's respective data dictionary, and metadata for each of the 23 surveyed plots can be found within the "Location_metadata.csv" file. While the "leafRaw.csv" and the "leafSmooth.csv" files contain all data for all species, the "leafPhenophaseDates2019-2023.csv" file currently excludes the dates for evergreen species in 2023. Another version of the file will be added as those dates are extracted.
Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.
The ESS-DIVE reporting format for file-level metadata (FLMD) provides granular information at the data file level to describe the contents, scope, and structure of the data file to enable comparison of data files within a data package. The FLMD are fully consistent with and augment the metadata collected at the data package level. We developed the FLMD template based on a review of a small number of existing FLMD in use at other agencies and repositories with valuable input from the Environmental Systems Science (ESS) Community. Also included is a template for a CSV Data Dictionary where users can provide file-level information about the contents of a CSV data file (e.g., define column names, provide units). Files are in .csv, .xlsx, and .md. Templates are in both .csv and .xlsx (open with e.g. Microsoft Excel, LibreOffice, or Google Sheets). Open the .md files by downloading and using a text editor (e.g. Notepad or TextEdit). Though we provide Excel templates for the file-level metadata reporting format, our instructions encourage users to 'Save the FLMD template as a CSV following the CSV Reporting Format guidance'. In addition, we developed the ESS-DIVE File Level Metadata Extractor which is a lightweight python script that can extract some FLMD fields following the recommended FLMD format and structure.
A master metadata repository (MMR) software system manages the storage and searching of metadata pertaining to data from national and international satellite sources of the Global Ocean Data Assimilation Experiment (GODAE) High Resolution Sea Surface Temperature Pilot Project [GHRSSTPP]. These sources produce a total of hundreds of data files daily, each file classified as one of more than ten data products representing global sea-surface temperatures. The MMR is a relational database wherein the metadata are divided into granulelevel records [denoted file records (FRs)] for individual satellite files and collection-level records [denoted data set descriptions (DSDs)] that describe metadata common to all the files from a specific data product. FRs and DSDs adhere to the NASA Directory Interchange Format (DIF). The FRs and DSDs are contained in separate subdatabases linked by a common field. The MMR is configured in MySQL database software with custom Practical Extraction and Reporting Language (PERL) programs to validate and ingest the metadata records. The database contents are converted into the Federal Geographic Data Committee (FGDC) standard format by use of the Extensible Markup Language (XML). A Web interface enables users to search for availability of data from all sources.