SEARCH · Engineering Papers

Results for “data diversity”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27

All sources NASA NTRS DOE OSTI

NASA NTRS2017

Models of Sector Aircraft Counts in the Presence of Local, Regional and Airport Constraints

Recently, the ATM community has made important progress in collaborative trajectory management through the introduction of a new FAA traffic management initiative called a Collaborative Trajectory Options Program (CTOP). FAA can use CTOPs to manage air traffic under multiple constraints (manifested as flow constrained areas or FCAs) in the system, and it allows flight operators to indicate their preferences for routing and delay options. CTOPs also permits better management of the overall trajectory of flights by considering both routing and departure delay options simultaneously. However, adoption of CTOPs in airspace has been hampered by many factors that include challenges in how to identify constrained areas and how to set rates for the FCAs. Decision support tools providing assistance would be particularly helpful in effective use of CTOPs. Such DSTs tools would need models of demand and capacity in the presence of multiple constraints. This study examines different approaches to using historical data to create and validate models of maximum flows in sectors and other airspace regions in the presence of multiple constraints. A challenge in creating an empirical model of flows under multiple constraints is a lack of sufficient historical data that captures diverse situations involving combinations of multiple constraints especially those with severe weather. The approach taken here to deal with this is two-fold. First, we create a generalized sector model encompassing multiple sectors rather than individual sectors in order to increase the amount of data used for creating the model by an order of magnitude. Secondly, we decompose the problem so that the amount of data needed is reduced. This involves creating a baseline demand model plus a separate weather constrained flow reduction model and then composing these into a single integrated model. A nominal demand model is a flow model (gdem) in the presence of clear local weather. This defines the flow as a function of weather constraints in neighboring regions, airport constraints and weather in locations that can cause re-routes to the location of interest. A weather constrained flow reduction model (fwx-red) is a model of reduction in baseline counts as a function of local weather. Because the number of independent variables associated with each of the two decomposed models is smaller than that with a single model, need for amount of data is reduced. Finally, a composite model that combines these two can be represented as fwx-red (gdem(e), l) where e represents non-local constraints and l represents local weather. The approaches studied to developing these models are divided into three categories: (1) Point estimation models (2) Empirical models (3) Theoretical models. Errors in predictions of these different types of models have been estimated. In situations when there is abundant data, point estimation models tend to be very accurate. In contrast, empirical models do better than theoretical models when there is some data available. The biggest benefit of theoretical models is their general applicability in wider range situations once the degree of accuracy of these has been established.

Kulkarni, Deepak↗

DOE OSTI2021

Data from Raczka et al., Interactions between microbial diversity and substrate chemistry determine the fate of carbon in soil, from Elizabeth Woods in Morgantown, West Virginia, USA

File types include: Soil respiration as 13C cumulative respiration from soil incubation experiment, qSIP (quantitative stable isotope probing) data that is "18S apes wide" which is the fungal atom percent excess or how much the fungi took up of the labeled litter and "16S apes wide" which is the bacterial atom percent excess or how much the bacteria took up of the labeled litter, metabolomic data from the incubated soil samples called "Metab Soil Unique Filtered", which is the metabolomic data that had an initial filter from the raw data, and lipidomic data from the incubated soil samples called "All lipids soil". These files were used according to methods in Raczka et al. 2021 Scientific Reports. Headers show soil type (Oak, Poplar) and litter type (also called substrate here, which is labeled as no litter (control), poplar, and oak). This research was done to understand interactions between ecosystem-level processes and microbial diversity that impacts the composition of decomposition products that can form stable soil organic matter.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

Point-spread Function Deconvolution of the IFU Data and Restoration of Galaxy Stellar Kinematics

We present a performance test of the point-spread function (PSF) deconvolution algorithm applied to astronomical integral field unit (IFU) spectroscopy data for restoration of galaxy kinematics. We deconvolve the IFU data by applying the Lucy–Richardson algorithm to the 2D image slice at each wavelength. We demonstrate that the algorithm can effectively recover the true stellar kinematics of the galaxy, by using mock IFU data with a diverse combination of surface brightness profile, signal-to-noise ratio, line-of-sight geometry, and line-of-sight velocity distribution (LOSVD). In addition, we show that the proxy of the spin parameter ${\lambda }_{{R}_{e}}$ can be accurately measured from the deconvolved IFU data. We apply the deconvolution algorithm to the actual SDSS-IV MaNGA IFU survey data. The 2D LOSVD, geometry, and ${\lambda }_{{R}_{e}}$ measured from the deconvolved MaNGA IFU data exhibit noticeable differences compared to the ones measured from the original IFU data. The method can be applied to any other regular-grid IFU data to extract the PSF-deconvolved spatial information.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2019

Identifying emerging phenomenon in long temporal phenotyping experiments

Abstract Motivation The rapid improvement of phenotyping capability, accuracy and throughput have greatly increased the volume and diversity of phenomics data. A remaining challenge is an efficient way to identify phenotypic patterns to improve our understanding of the quantitative variation of complex phenotypes, and to attribute gene functions. To address this challenge, we developed a new algorithm to identify emerging phenomena from large-scale temporal plant phenotyping experiments. An emerging phenomenon is defined as a group of genotypes who exhibit a coherent phenotype pattern during a relatively short time. Emerging phenomena are highly transient and diverse, and are dependent in complex ways on both environmental conditions and development. Identifying emerging phenomena may help biologists to examine potential relationships among phenotypes and genotypes in a genetically diverse population and to associate such relationships with the change of environments or development. Results We present an emerging phenomenon identification tool called Temporal Emerging Phenomenon Finder (TEP-Finder). Using large-scale longitudinal phenomics data as input, TEP-Finder first encodes the complicated phenotypic patterns into a dynamic phenotype network. Then, emerging phenomena in different temporal scales are identified from dynamic phenotype network using a maximal clique based approach. Meanwhile, a directed acyclic network of emerging phenomena is composed to model the relationships among the emerging phenomena. The experiment that compares TEP-Finder with two state-of-art algorithms shows that the emerging phenomena identified by TEP-Finder are more functionally specific, robust and biologically significant. Availability and implementation The source code, manual and sample data of TEP-Finder are all available at: http://phenomics.uky.edu/TEP-Finder/. Supplementary information Supplementary data are available at Bioinformatics online.

Biochemistry & Molecular Biology↗

NASA NTRS2016

Developing the Next Generation of Science Data System Engineers

At Goddard, engineers and scientists with a range of experience in science data systems are needed to employ new technologies and develop advances in capabilities for supporting new Earth and Space science research. Engineers with extensive experience in science data, software engineering and computer-information architectures are needed to lead and perform these activities. The increasing types and complexity of instrument data and emerging computer technologies coupled with the current shortage of computer engineers with backgrounds in science has led the need to develop a career path for science data systems engineers and architects.The current career path, in which undergraduate students studying various disciplines such as Computer Engineering or Physical Scientist, generally begins with serving on a development team in any of the disciplines where they can work in depth on existing Goddard data systems or serve with a specific NASA science team. There they begin to understand the data, infuse technologies, and begin to know the architectures of science data systems. From here the typical career involves peermentoring, on-the-job training or graduate level studies in analytics, computational science and applied science and mathematics. At the most senior level, engineers become subject matter experts and system architect experts, leading discipline-specific data centers and large software development projects. They are recognized as a subject matter expert in a science domain, they have project management expertise, lead standards efforts and lead international projects. A long career development remains necessary not only because of the breadth of knowledge required across physical sciences and engineering disciplines, but also because of the diversity of instrument data being developed today both by NASA and international partner agencies and because multidiscipline science and practitioner communities expect to have access to all types of observational data.This paper describes an approach to defining career-path guidance for college-bound high school and undergraduate engineering students, junior and senior engineers from various disciplines.

Next Generation of Science↗

DOE OSTI2025

Development of a multiphase equation of state for gallium with experiments and ab initio free-energy calculations

Here, we present a five-phase equation of state (EOS) for elemental gallium (Ga) that is developed using both experimental data and new theoretical predictions. Four experimentally observed solid phases (Ga-I, Ga-II, Ga-III, and Ga-IV) and one liquid phase are included. To improve our understanding of the thermal behavior of Ga and its phase boundaries under compression, we have performed ab initio density functional theory (DFT) free-energy calculations for the Ga-III and liquid phases, which enables us to determine the melt temperature to be 2214 ± 100 K at 110 GPa, extending significantly beyond the existing experimental melt data, which are limited to 25 GPa. In order to best describe the electron-thermal contribution, which dominates the liquid free energy at high temperatures, we have carried out averaged-atom-in-jellium DFT calculations to cover the entire temperature and density range of the EOS. The resulting multiphase Ga EOS is able to accurately reproduce a diverse variety of data, including known phase boundaries, the principal Hugoniot, low-pressure liquid isobars, and diamond-anvil-cell isotherm measurements at high pressures. It agrees more closely with key experimentally measured properties than other Ga EOS models targeted for high-pressure applications.

Wu, Christine J. [Lawrence Livermore National Labo↗

DOE OSTI2018

FRAMES Metadata Reporting Templates for Ecohydrological Observations, version 1.1

FRAMES is a a set of Excel metadata files and package-level descriptive metadata that are designed to facilitate and improve capture of desired metadata for ecohydrological observations. The metadata are bundled with data files into a data package and submitted to a data repository (e.g. the NGEE Tropics Data Repository) via a web form. FRAMES standardizes reporting of diverse ecohydrological and biogeochemical data for synthesis across a range of spatiotemporal scales and incorporates many best data science practices. This version of FRAMES supports observations for primarily automated measurements collected by permanently located sensors, including sap flow (tree water use), leaf surface temperature, soil water content, dendrometry (stem diameter growth increment), and solar radiation. Version 1.1 extend the controlled vocabulary and incorporates functionality to facilitate programmatic use of data and FRAMES metadata (R code available at NGEE Tropics Data Repository).

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2016

Population genomic analysis reveals differential evolutionary histories and patterns of diversity across subgenomes and subpopulations of Brassica napus L.

The allotetraploid species Brassica napus L. is a global crop of major economic importance, providing canola oil (seed) and vegetables for human consumption and fodder and meal for livestock feed. Characterizing the genetic diversity present in the extant germplasm pool of B. napus is fundamental to better conserve, manage and utilize the genetic resources of this species. We used sequence-based genotyping to identify and genotype 30,881 SNPs in a diversity panel of 782 B. napus accessions, representing samples of winter and spring growth habits originating from 33 countries across Europe, Asia, and America. We detected strong population structure broadly concordant with growth habit and geography, and identified three major genetic groups: spring (SP), winter Europe (WE), and winter Asia (WA). Subpopulation-specific polymorphism patterns suggest enriched genetic diversity within the WA group and a smaller effective breeding population for the SP group compared to WE. Interestingly, the two subgenomes of B. napus appear to have different geographic origins, with phylogenetic analysis placing WE and WA as basal clades for the other subpopulations in the C and A subgenomes, respectively. Finally, we identified 16 genomic regions where the patterns of diversity differed markedly from the genome-wide average, several of which are suggestive of genomic inversions. The results obtained in this study constitute a valuable resource for worldwide breeding efforts and the genetic dissection and prediction of complex B. napus traits.

58 GEOSCIENCES↗

NASA NTRS2011

Towards a Standard for Provenance and Context for Preservation of Data for Earth System Science

Long-term data sets with data from many missions are needed to study trends and validate model results that are typical in Earth System Science research. Data and derived products originate from multiple missions (spaceborne, airborne and/or in situ) and from multiple organizations. During the missions as well as well past their termination, it is essential to preserve the data and products to support future studies. Key aspects of preservation are: preserving bits and ensuring data are uncorrupted, preserving understandability with appropriate documentation, and preserving reproducibility of science with appropriate documentation and other artifacts. Computer technology provides adequate standards to ensure that, with proper engineering, bits are preserved as hardware evolves. However, to ensure understandability and reproducibility, it is essential to plan ahead to preserve all the relevant data and information. There are currently no standards to identify the content that needs to be preserved, leading to non-uniformity in content and users not being sure of whether preserved content is comprehensive. Each project, program or agency can specify the items to be preserved as a part of its data management requirements. However, broader community consensus that cuts across organizational or national boundaries would be needed to ensure comprehensiveness, uniformity and long-term utility of archived data. The Federation of Earth Science Information Partners (ESIP), a diverse network of scientists, data stewards and technology developers, has a forum for ESIP members to collaborate on data preservation issues. During early 2011, members discussed the importance of developing a Provenance and Context Content Standard (PCCS) and developed an initial list of content items. This list is based on the outcome of a NASA and NOAA meeting held in 1998 under the auspices of the USGCRP, documentation requirements from NOAA and our experience with some of the NASA Earth science missions. The items are categorized into the following 8 high level categories: Preflight/Pre-Operations, Products (Data), Product Documentation, Mission Calibration, Product Software, Algorithm Input, Validation, Software Tools.

Ramaprian, Hampapuram K.↗

DOE OSTI2021

Interpreting the Lipidome: Bioinformatic Approaches to Embrace the Complexity

Background Improvements in mass spectrometry (MS) technologies coupled with bioinformatics developments have allowed considerable advancement in the measurement and interpretation of lipidomics data in recent years. Since research areas employing lipidomics are rapidly increasing, there is a great need for bioinformatic tools that capture and utilize the complexity of the data. Currently, the diversity and complexity within the lipidome is often concealed by summing over or averaging individual lipids up to (sub)class-based descriptors, losing valuable information about biological function and interactions with other distinct lipids molecules, proteins and/or metabolites. Aim of review To address this gap in knowledge, novel bioinformatics methods are needed to improve identification, quantification, integration and interpretation of lipidomics data. The purpose of this mini-review is to summarize exemplary methods to explore the complexity of the lipidome. Key scientific concepts of review Here we describe six approaches that capture three core focus areas for lipidomics: (1) lipidome annotation including a resolvable database identifier, (2) interpretation via pathway- and enrichment-based methods, and (3) understanding complex interactions to emphasize specific steps in the analytical process and highlight challenges in analyses associated with the complexity of lipidome data.

Kyle, Jennifer E.↗

DOE OSTI2011

Molecular Assemblies, Genes and Genomics Integrated Efficiently (MAGGIE)

Final report on MAGGIE. We set ambitious goals to model the functions of individual organisms and their community from molecular to systems scale. These scientific goals are driving the development of sophisticated algorithms to analyze large amounts of experimental measurements made using high throughput technologies to explain and predict how the environment influences biological function at multiple scales and how the microbial systems in turn modify the environment. By experimentally evaluating predictions made using these models we will test the degree to which our quantitative multiscale understanding wilt help to rationally steer individual microbes and their communities towards specific tasks. Towards this end we have made substantial progress towards understanding evolution of gene families, transcriptional structures, detailed structures of keystone molecular assemblies (proteins and complexes), protein interactions, biological networks, microbial interactions, and community structure. Using comparative analysis we have tracked the evolutionary history of gene functions to understand how novel functions evolve. One level up, we have used proteomics data, high-resolution genome tiling microarrays, and 5' RNA sequencing to revise genome annotations, discover new genes including ncRNAs, and map dynamically changing operon structures of five model organisms: For Desulfovibrio vulgaris Hildenborough, Pyrococcus furiosis, Sulfolobus solfataricus, Methanococcus maripaludis and Haiobacterium salinarum NROL We have developed machine learning algorithms to accurately identify protein interactions at a near-zero false positive rate from noisy data generated using tagfess complex purification, TAP purification, and analysis of membrane complexes. Combining other genome-scale datasets produced by ENIGMA (in particular, microarray data) and available from literature we have been able to achieve a true positive rate as high as 65% at almost zero false positives when applied to the manually curated training set. Applying this method to the data representing around a quarter of the fraction space for water soluble proteins in D. vulgaris, we obtained 854 reliable pair wise interactions. Further, we have developed algorithms to analyze and assign significance to protein interaction data from bait pull-down experiments and integrate these data with other systems biology data through associative biclustering in a parallel computing environment. We will 'fill-in' missing information in these interaction data using a 'Transitive Closure' algorithm and subsequently use 'Between Commonality Decomposition' algorithm to discover complexes within these large graphs of protein interactions. To characterize the metabolic activities of proteins and their complexes we are developing algorithms to deconvolute pure mass spectra, estimate chemical formula for m/z values, and fit isotopic fine structure to metabolomics data. We have discovered that in comparison to isotopic pattern fitting methods restricting the chemical formula by these two dimensions actually facilitates unique solutions for chemical formula generators. To understand how microbial functions are regulated we have developed complementary algorithms for reconstructing gene regulatory networks (GRNs). Whereas the network inference algorithms cMonkey and Inferelator developed enable de novo reconstruction of predictive models for GRNs from diverse systems biology data, the RegPrecise and RegPredict framework developed uses evolutionary comparisons of genomes from closely related organisms to reconstruct conserved regulons. We have integrated the two complementary algorithms to rapidly generate comprehensive models for gene regulation of understudied organisms. Our preliminary analyses of these reconstructed GRNs have revealed novel regulatory mechanisms and cis-regulatory motifs, as well asothers that are conserved across species. Finally, we are supporting scientific efforts in ENIGMA with data management solutions and by integrating all of the algorithms, software and data into a Knowledgebase. For instance, we have developed the RegPrecise database (http://regprecise.lbl.gov) which represents manually curated sets of regulons laying the basis for automatic annotation of regulatory interactions in closely related species. We are also in the midst of scaling up MicrobesOnline to handle the growing volume of sequence and functional genomics data. Over the last year our efforts have been focused on providing support for additional genomic and functional genomic data types. Similarly, we have developed several visualization tools to help with the exploration of complex systems biology datasets. A case in point is the Gaggle Genome Browser (GGB), which was enhanced with visualizations for plotting peptide detections and protein-DNA binding alongside transcriptome structure, plus the ability to interactively filter by signal intensity or p-value.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

A database and meta-analysis on the performance of exploding pusher implosions conducted at OMEGA

A database of 222 exploding pusher implosions conducted at the OMEGA Laser Facility is presented. The dataset consists of glass-shell capsules filled with varying pressures of D 2 , T 2 , and 3 He, which were imploded using square laser pulses with intensities ranging from 1 to 1 × 10 15 W/cm 2 . The database includes measurements of bang times, ion temperatures, and yields from the DD, D 3 He, and DT fusion reactions. A semi-analytic exploding pusher model is introduced, which effectively captures the observed trends in the data. This model predicts that the measurements scale according to a power-law relation based on the initial capsule and laser conditions. A generalized power-law scaling relation is directly fit to each dataset, providing a useful interpolation of the entire database. Overall, the database provides a valuable resource to estimating bang times, temperatures, and yields for the design of future experiments. Additionally, it provides a diverse set of data for validating more advanced implosion physics models.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2011

Building the International Lattice Data Grid

We present the International Lattice Data Grid (ILDG), a loosely federated grid-of-grids for sharing data from Lattice Quantum Chromodynamics (LQCD) simulations. The ILDG comprises of metadata, file-format and web-service standards, which can be used to wrap regional data-grid interfaces, allowing seamless access to catalogues and data in a diverse set of collaborating regional grids. We discuss the technological underpinnings of the ILDG, primarily the metadata and the middleware, and offer a critique of its various aspects with the hindsight of the design work and the two years of production.

99 GENERAL AND MISCELLANEOUS↗

NASA NTRS2019

Leveraging STARE for Co-aligned Data Locality with netCDF and Python MPI

We have leveraged STARE indexing to package partitioned data chunks from diverse datasets into netCDF files, distributed them on a cluster of 16 lightweight nodes with their placements spatiotemporally co-aligned, and demonstrated a few integrative analyses using netCDF parallel I/O and Python MPI, with single-user performance and scalability comparable to, or even better than, that of a parallel array database management system (ADBMS) such as SciDB. However, records of the node location and STARE index ranges for each data chunk, similar to the chunk maps of SciDB, must be maintained and consulted by the I/O and analysis code for coordinating the analytic operations in parallel, in order to achieve the good performance and scalability.

Kwo-Sen Kuo↗

NASA NTRS1991

The relation between FIR and H I emission in galaxies

This project explored the relationship between the global far-infrared and neutral hydrogen (H I) emission from galaxies, based on data from the Infrared Astronomy Satellite (IRAS) and published radio data. 100 and 60 micron IRAS fluxes were used to establish a temperature corrected measure of the cold dust emission, and H I fluxes were drawn from the literature with the greatest possible consistency. The degree of correlation between the FIR and H I fluxes was found to be better than in previous studies, comparable to the correlation previously found between FIR and CO fluxes. The improvement was obtained largely by (1) separating 'stripped' from 'unstripped' galaxies, and (2) using compatible sources of H I data. Stripping occurs in clusters of galaxies and is probably caused by ram-pressure effects as a galaxy travels through the intergalactic medium. Our results suggest that stripped galaxies have had their outer-disk gas removed (approximately 80% of their total H I) while retaining most of their 100-micron-emitting dust. This strongly shifts the ratio of their 100-micron-to-H I fluxes. The second problem, arising from diverse sources of data, arises because differing telescopes and observational techniques give rise to substantial disagreement in the measured H I flux, and this degrades the correlation of the FIR and H I fluxes.

Schneider, Stephen E.↗

DOE OSTI2020

Extinction Dynamics of the Foot-and-Mouth Disease Virus Carrier State Under Natural Conditions

Foot-and-mouth disease (FMD) is one of the most economically important livestock diseases worldwide. Following the clinical phase of FMD, a large proportion of ruminants remain persistently infected for extended periods. Although extinction of this carrier state occurs continuously at the animal and population levels, studies vary widely in their estimates of the duration of persistent infection. There is a need for robust statistical models to capture the dynamics of persistent infection for the sake of guiding FMD control and trade policies. The goal of the current study was to develop and assess statistical models to describe the extinction of FMD virus (FMDV) persistent infection using data from primary longitudinal studies of naturally infected cattle and Asian buffalo in Vietnam and India. Specifically, accelerated failure time (AFT) models and generalized linear mixed models (GLMM) were developed to predict the probability of persistent infection in seropositive animals and identified carriers at the individual animal level at sequential time points after outbreaks. The primary studies were analyzed by country and combined using an individual-participant data meta-analysis approach. The models estimated similar trends in the duration of persistent infection for the study/species groups included in the analyses, however the significance of the trends differed between the models. The overall probabilities of persistent infection were similar as predicted by the AFT and GLMM models: 6 months: 99% (AFT) /80% (GLMM), 12 months: 51% (AFT) /32% (GLMM), 18 months: 6% (AFT) /5% (GLMM), 24 months: 0.8% (AFT) /0.6% (GLMM). These models utilizing diverse and robust data sets predict higher probabilities of persistence than previously published, suggesting greater endurance of carriers subsequent to an outbreak. This study demonstrates the utility of statistical models to investigate the dynamics of persistent infection and the importance of large datasets, which can be achieved by combining data from several smaller studies in meta-analyses. Results of this study enhance current knowledge of the FMDV carrier state and may inform policy decisions regarding FMDV persistent infection.

59 BASIC BIOLOGICAL SCIENCES↗

NASA NTRS2007

ShareSync: A Solution for Deterministic Data Sharing over Ethernet

As part of upgrading the Contact Dynamics Simulation Laboratory (CDSL) at the NASA Marshall Space Flight Center (MSFC), a simple, cost effective method was needed to communicate data among the networked simulation machines and I/O controllers used to run the facility. To fill this need and similar applicable situations, a generic protocol was developed, called ShareSync. ShareSync is a lightweight, real-time, publish-subscribe Ethernet protocol for simple and deterministic data sharing across diverse machines and operating systems. ShareSync provides a simple Application Programming Interface (API) for simulation programmers to incorporate into their code. The protocol is compatible with virtually all Ethernet-capable machines, is flexible enough to support a variety of applications, is fast enough to provide soft real-time determinism, and is a low-cost resource for distributed simulation development, deployment, and maintenance. The first design cycle iteration of ShareSync has been completed, and the protocol has undergone several testing procedures including endurance and benchmarking tests and approaches the 2001ts data synchronization design goal for the CDSL.

Dunn, Daniel J., II↗

DOE OSTI2019

A Survey of Intrusion Detection Systems Leveraging Host Data

This survey focuses on intrusion detection systems (IDS) that leverage host-based data sources for detecting attacks on enterprise network. The host-based IDS (HIDS) literature is organized by the input data source, presenting targeted sub-surveys of HIDS research leveraging system logs, audit data, Windows Registry, file systems, and program analysis. While system calls are generally included in audit data, several publicly available system call datasets have spawned a flurry of IDS research on this topic, which merits a separate section. To accommodate current researchers, a section giving descriptions of publicly available datasets is included, outlining their characteristics and shortcomings when used for IDS evaluation. Related surveys are organized and described. All sections are accompanied by tables concisely organizing the literature and datasets discussed. Finally, challenges, trends, and broader observations are throughout the survey and in the conclusion along with future directions of IDS research. Overall, this survey was designed to allow easy access to the diverse types of data available on a host for sensing intrusion, the progressions of research using each, and the accessible datasets for prototyping in the area.

97 MATHEMATICS AND COMPUTING↗