Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data diversity”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

MAPPRAISER: A massively parallel map-making framework for multi-kilo pixel CMB experiments

Forthcoming cosmic microwave background (CMB) polarized anisotropy experiments have the potential to revolutionize our understanding of the Universe and fundamental physics. The sought-after, tale-telling signatures will be however distributed over voluminous data sets which these experiments will collect. These data sets will need to be efficiently processed and unwanted contributions due to astrophysical, environmental, and instrumental effects characterized and efficiently mitigated in order to uncover the signatures. This poses a significant challenge to data analysis methods, techniques, and software tools which will not only have to be able to cope with huge volumes of data but to do so with unprecedented precision driven by the demanding science goals posed for the new experiments. A keystone of efficient CMB data analysis is solvers of very large linear systems of equations. Such systems appear in very diverse contexts throughout CMB data analysis pipelines, however they typically display similar algebraic structures and can therefore be solved using similar numerical techniques. Linear systems arising in the so-called map-making problem are one of the most prominent and common ones. In this work we present a massively parallel, flexible and extensible framework, comprised of a numerical library, MIDAPACK, and a high level code, MAPPRAISER, which provide tools for solving efficiently such systems. Here, the framework implements iterative solvers based on conjugate gradient techniques: enlarged and preconditioned using different preconditioners. We demonstrate the framework on simulated examples reflecting basic characteristics of the forthcoming data sets issued by ground-based and satellite-borne instruments, executing it on as many as 16,384 compute cores. The software is developed as an open source project freely available to the community at: https://github.com/B3Dcmb/midapack.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS1973

Handling properties of diverse automobiles and correlation with full scale response data

Driver/vehicle response and performance of a variety of vehicles in the presence of aerodynamic disturbances are discussed. Steering control is emphasized. The vehicles include full size station wagon, sedan, compact sedan, van, pickup truck/camper, and wagon towing trailer. Driver/vehicle analyses are used to estimate response and performance. These estimates are correlated with full scale data with test drivers and the results are used to refine the driver/vehicle models, control structure, and loop closure criteria. The analyses and data indicate that the driver adjusts his steering control properties (when he can) to achieve roughly the same level of performance despite vehicle variations. For the more disturbance susceptible vehicles, such as the van, the driver tightens up his control. Other vehicles have handling dynamics which cause him to loosen his control response, even though performance degrades.

Hoh, R. H.↗
DOE OSTI2017

Pattern Discovery in Internet Background Radiation

Internet Background Radiation (IBR) is observed in empty network address spaces. No traffic should arrive there, but it does in overwhelming quantities, gathering evidences of attacks, malwares and misconfigurations. The study of IBR helps to detect spreading network problems, common vulnerabilities and attack trends. However, network traffic data evolves quickly and is of high volume and diversity, i.e., an outstanding big data challenge. When used to assist network security, it also requires the online classification of dynamic streaming data. In this paper, we introduce an AGgregation & Mode (AGM) vector to represent network traffic. The AGM format characterizes IP hosts by extracting aggregated and mode values of IP header fields, and without inspecting payloads. We performed clustering and statistical analysis to explore six months of IBR from 2012 with the AGM mapping. The discovered patterns allow building a classification of IBR, which identifies phenomena that have been actively polluting the Internet for years. The AGM representation is light and tailored for monitoring and pattern discovery. We show that AGM vectors are suitable to analyze large volumes of network traffic: they capture permanent operations, such as long term scanning, as well as bursty events from targeted attacks and short term incidents.

Iglesias, Felix↗
NASA NTRS1990

Data analysis techniques

A large and diverse number of computational techniques are routinely used to process and analyze remotely sensed data. These techniques include: univariate statistics; multivariate statistics; principal component analysis; pattern recognition and classification; other multivariate techniques; geometric correction; registration and resampling; radiometric correction; enhancement; restoration; Fourier analysis; and filtering. Each of these techniques will be considered, in order.

Park, Steve↗
NASA NTRS1996

Multimode Data-Compression System

Data-compression system developed to satisfy need for high-speed, high-performance compression of data from sources as diverse as medical images, high-definition television images, audio signals, readouts from scientific instruments, and binary data files. Maximum data-transmission capability of communication channel or storage capacity of storage device multiplied by approximately compression ratio. Various combinations of lossless and lossy compression chosen to suit various data streams.

Fang, Wai-Chi↗
DOE OSTI2018

Nonpareil 3: Fast Estimation of Metagenomic Coverage and Sequence Diversity

Estimations of microbial community diversity based on metagenomic data sets are affected, often to an unknown degree, by biases derived from insufficient coverage and reference database-dependent estimations of diversity. For instance, the completeness of reference databases cannot be generally estimated since it depends on the extant diversity sampled to date, which, with the exception of a few habitats such as the human gut, remains severely undersampled. Further, estimation of the degree of coverage of a microbial community by a metagenomic data set is prohibitively time-consuming for large data sets, and coverage values may not be directly comparable between data sets obtained with different sequencing technologies. Here, we extend Nonpareil, a database-independent tool for the estimation of coverage in metagenomic data sets, to a high-performance computing implementation that scales up to hundreds of cores and includes, in addition, a k -mer-based estimation as sensitive as the original alignment-based version but about three hundred times as fast. Further, we propose a metric of sequence diversity ( N d ) derived directly from Nonpareil curves that correlates well with alpha diversity assessed by traditional metrics. We use this metric in different experiments demonstrating the correlation with the Shannon index estimated on 16S rRNA gene profiles and show that N d additionally reveals seasonal patterns in marine samples that are not captured by the Shannon index and more precise rankings of the magnitude of diversity of microbial communities in different habitats. Therefore, the new version of Nonpareil, called Nonpareil 3, advances the toolbox for metagenomic analyses of microbiomes.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2017

First results from the TUS orbital detector in the extensive air shower mode

TUS (Tracking Ultraviolet Set-up), the first orbital detector of extreme energy cosmic rays (EECRs), those with energies above 50 EeV, was launched into orbit on April 28, 2016, as a part of the Lomonosov satellite scientific payload. The main aim of the mission is to test a technique of registering fluorescent and Cherenkov radiation of extensive air showers generated by EECRs in the atmosphere with a space telescope. We present preliminary results of its operation in a mode dedicated to registering extensive air showers in the period from August 16, 2016, to November 4, 2016. No EECRs have been conclusively identified in the data yet, but the diversity of ultraviolet emission in the atmosphere was found to be unexpectedly rich. We discuss typical examples of data obtained with TUS and their possible origin. The data is important for obtaining more accurate estimates of the nocturnal ultraviolet glow of the atmosphere, necessary for successful development of more advanced orbital EECR detectors including those of the KLYPVE (K-EUSO) and JEM-EUSO missions.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Classifying and analyzing small-angle scattering data using weighted k nearest neighbors machine learning techniques

A consistent challenge for both new and expert practitioners of small-angle scattering (SAS) lies in determining how to analyze the data, given the limited information content of said data and the large number of models that can be employed. Machine learning (ML) methods are powerful tools for classifying data that have found diverse applications in many fields of science. Here, ML methods are applied to the problem of classifying SAS data for the most appropriate model to use for data analysis. The approach employed is built around the method of weighted k nearest neighbors (wKNN), and utilizes a subset of the models implemented in the SasView package (https://www.sasview.org/) for generating a well defined set of training and testing data. The prediction rate of the wKNN method implemented here using a subset of SasView models is reasonably good for many of the models, but has difficulty with others, notably those based on spherical structures. A novel expansion of the wKNN method was also developed, which uses Gaussian processes to produce local surrogate models for the classification, and this significantly improves the classification accuracy. Further, by integrating a stochastic gradient descent method during post-processing, it is possible to leverage the local surrogate model both to classify the SAS data with high accuracy and to predict the structural parameters that best describe the data. The linking of data classification and model fitting has the potential to facilitate the translation of measured data into results for both novice and expert practitioners of SAS.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Working Group on Virtual Data Integration

This report is the outcome of a workshop commissioned by the U.S. Department of Energy’s (DOE) Climate and Environmental Sciences Division (CESD) to examine current and future data infrastructure requirements foundational for achieving CESD scientific mission goals in advancing a robust, predictive understanding of Earth’s climate and environmental systems. Over the past several years, data volumes in CESD disciplines have risen sharply to unprecedented levels (tens of petabytes). Moreover, the complexity and diversity of this research data— including simulations, observations, and reanalysis— have grown significantly, posing new challenges for data capture, storage, verification, analysis, and integration. With the trends of increased data volume (in the hundreds of petabytes), more complex analysis processes, and growing cross-disciplinary collaborations, it is timely to investigate whether the CESD community has the computational and data support needed to fully realize the scientific potential of its data collections. In recognition of the challenges, a partnership is forming across CESD and among national and international agencies to examine the viability of creating an integrated, collaborative data infrastructure: a Virtual Laboratory. The overarching goal of this report is to identify the community’s key data technology requirements and high-priority development needs for sustaining and growing its scientific discovery potential. The report also aims to map these requirements to existing solutions and to identify gaps in current services, tools, and infrastructure that will need to be addressed in the short, medium, and long term to advance scientific progress.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2014

The Near Earth Object Scout Spacecraft: A Low Cost Approach to in-situ Characterization of the NEO Population

In this paper we describe a micro/nano satellite spacecraft and a supporting mission profile and architecture designed to enable preliminary in-situ characterization of a significant number of Near Earth Objects (NEOs) at reasonable cost. The spacecraft will be referred to as the NEO Scout. NEO Scout spacecraft are to be placed in GTO, GEO, or cis-lunar space as secondary payloads on launch vehicles headed for GTO or beyond and will begin their mission after deployment from the launcher. A distinguishing key feature of the NEO scout system is to design the mission timeline and spacecraft to rendezvous with and land on the target NEOs during close approach to the Earth-Moon system using low-thrust/high- impulse propulsion systems. Mission feasibility and preliminary design analysis are presented along with detailed trajectory calculations. The use of micro/nano satellites in low-cost interplanetary exploration is attracting increasing attention and is the subject of several annual workshops and published design studies (1-4). The NEO population consists of those asteroids and short period comets orbiting the Sun with a perihelion of 1.3 astronomical units or less (5-8). As of July 30, 2013 10065 Near-Earth objects have been discovered. The spin rate, mass, density, surface physical (especially mechanical) properties, composition, and mineralogy of the vast majority of these objects are highly uncertain and the limited available telescopic remote sensing data imply a very diverse population (5-8). In-situ measurements by robotic spacecraft are urgently needed to provide the characterization data needed to support hardware and mission design for more ambitious human and robotic NEO operations. Large numbers of NEOs move into close proximity with the Earth-Moon system every year (9). The JPL Near-Earth Object Human Space Flight Accessible Targets Study (NHATS) (10) has produced detailed mission profile and delta V requirements for various NEO missions ranging from 30 to 420 days in duration and assuming chemical propulsion. Similar studies have been reported assuming high power electric propulsion for manned NEO rendezvous missions (11). The delta V requirement breakdown and mission profile data from references 10 and 11 are used as a basis for sizing the NEO Scout spacecraft and for conducting preliminary feasibility assessments using the Tsiokolvsky rocket equation, a (worst-case) delta V requirement of 10 km/sec, and a maximum spacecraft dry mass of 20 kg. Using chemical propellant for a 10 km/sec delta V drives spacecraft wet mass well above 300 kg so that chemical propulsion is a non-starter for the proposed mission profile and spacecraft wet mass limits. In contrast, a solar electric propulsion system needs only 8 kg of Xe propellant to accelerate the spacecraft to 10 km/sec in 163 days with 0.02 N of thrust and 500 W of power from1.6 sq m of 29% efficient solar panels. In a second example, accelerating a 4 kg payload to 7 km/sec over 180 days requires about 6.7 kg of propellant and 1.2 kg of solar panels (12 kg total spacecraft wet mass).

Koontz, Steven L.↗
DOE OSTI2025

High-frequency Data Integration for Landscape Model Calibration of Carbon Fluxes Across Diverse Tidal Marshes

Terrestrial Aquatic Interfaces (TAIs), and tidal wetlands in particular, store large amounts of carbon yet are not well represented in Earth System Models (ESMs). Predictions of carbon cycling and greenhouse gas (GHG) emissions in tidal wetlands are highly uncertain. Eddy covariance (EC) towers provide ecosystem-scale GHG flux data at a temporal resolution (every 30min) that is helpful for parameterizing and improving mechanistic realism in ESMs. We propose to use a network of eddy covariance towers and standardized ancillary data streams, along with mesocosm experiments and statistical analyses, across diverse tidal wetlands of North America to develop and improve biogeochemical modeling at the TAI. Our overarching objective is to improve understanding and process-based modeling of gross primary productivity (GPP) and CH4 emission responses, both non-linear and asynchronous, to stressors including plant inundation, disturbance, salinity and nitrogen loading.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

PanFP: Pangenome-based functional profiles for microbial communities

For decades there has been increasing interest in understanding the relationships between microbial communities and ecosystem functions. Current DNA sequencing technologies allows for the exploration of microbial communities in two principle ways: targeted rRNA gene surveys and shotgun metagenomics. For large study designs, it is often still prohibitively expensive to sequence metagenomes at both the breadth and depth necessary to statistically capture the true functional diversity of a community. Although rRNA gene surveys provide no direct evidence of function, they do provide a reasonable estimation of microbial diversity, while being a very cost effective way to screen samples of interest for later shotgun metagenomic analyses. However, there is a great deal of 16S rRNA gene survey data currently available from diverse environments, and thus a need for tools to infer functional composition of environmental samples based on 16S rRNA gene survey data. As a result, we present a computational method called pangenome based functional profiles (PanFP), which infers functional profiles of microbial communities from 16S rRNA gene survey data for Bacteria and Archaea. PanFP is based on pangenome reconstruction of a 16S rRNA gene operational taxonomic unit (OTU) from known genes and genomes pooled from the OTU s taxonomic lineage. From this lineage, we derive an OTU functional profile by weighting a pangenome s functional profile with the OTUs abundance observed in a given sample. We validated our method by comparing PanFP to the functional profiles obtained from the direct shotgun metagenomic measurement of 65 diverse communities via Spearman correlation coefficients. These correlations improved with increasing sequencing depth, within the range of 0.8 0.9 for the most deeply sequenced Human Microbiome Project mock community samples. PanFP is very similar in performance to another recently released tool, PICRUSt, for almost all of survey data analysed here. But, our method is unique in that any OTU building method can be used, as opposed to being limited to closed reference OTU picking strategies against specific reference sequence databases. In conclusion, we developed an automated computational method, which derives an inferred functional profile based on the 16S rRNA gene surveys of microbial communities. The inferred functional profile provides a cost effective way to study complex ecosystems through predicted comparative functional metagenomes and metadata analysis. All PanFP source code and additional documentation are freely available online at GitHub.

16S rRNA survey↗
DOE OSTI2025

IRIS-MEMFLOW: Data Flow-Enabled Portable Memory Orchestration in IRIS Runtime for Diverse Heterogeneity

Task-based programming models and execution paradigms provide a means to decompose a computation by expressing it as a graph in which each node represents a specific computation operating on memory objects and the edges define the dependencies in the execution flow. In this execution model, independent nodes in the graph can be executed concurrently in different computing devices, making it suitable for heterogeneous systems in which computing devices with different architectures coexist. However, careful memory orchestration across heterogeneous devices is needed because copies of the same memory object may reside in multiple devices during execution. Manually ensuring such an orchestration is quite challenging. Not only must an application developer guard against race conditions, but they must also optimize data movement between the host and devices because unnecessary data movement significantly impacts performance. To mitigate these challenges, we enhance the IRIS heterogeneous runtime and introduce IRIS-MEMFLOW–a data flow–enabled portable memory abstraction for seamlessly orchestrating memory in diverse heterogeneous computing environments. By using data-flow analysis, IRIS-MEMFLOW guards against race conditions while multiple heterogeneous devices access memory objects. IRIS-MEMFLOW also optimizes data movement between the host and devices without manual intervention. As a result, IRIS provides improved programming productivity, performance, and portability for multidevice heterogeneous executions in high-performance computing and cloud systems that run diverse architectures from different vendors. The efficacy of IRIS-MEMFLOW is evaluated through experiments that show its capability in terms of programming productivity, multidevice heterogeneity, portability, and low overhead versus the state of the art.

Monil, M. A. H. [ORNL] (ORCID:0000000334194037)↗
NASA NTRS2012

Phytoplankton Assemblage Patterns in the Southern Mid-Atlantic Bight

As part of the Wallops Coastal Oceans Observing Laboratory (Wa-COOL) Project, we sampled a time-series transect in the southern Mid-Atlantic Bight (MAB) biweekly. Our 2-year time-series data included physical parameters, nutrient concentrations, and chlorophyll a concentrations. A detailed phytoplankton assemblage structure was examined in the second year. During the 2-year study, chlorophyll a concentration (and ocean color satellite imagery) indicated that phytoplankton blooms occurred in January/February during mixing conditions and in early autumn under stratified conditions. The chlorophyll a concentrations ranged from 0.25 microgram 1(exp -1) to 15.49 microgram 1(exp -1) during the 2-year period. We were able to discriminate approximately 116 different species under phase contrast microscopy. Dominant phytoplankton included Skeletonema costatum, Rhizosolenia spp., and Pseudo-nitzschia pungens. In an attempt to determine phytoplankton species competition/succession within the assemblage, we calculated a Shannon Weaver diversity index for our diatom microscopy data. Diatom diversity was greatest during the winter and minimal during the spring. Diatom diversity was also greater at nearshore stations than at offshore stations. Individual genera appeared patchy, with surface and subsurface patches appearing abruptly and persisting for only 1-2 months at a time. The distribution of individual species differed significantly from bulk variables of the assemblage (chlorophyll a ) and total phytoplankton assemblage (cells), which indicates that phytoplankton species may be limited in growth in ways that differ from those of the total assemblage. Our study demonstrated a highly diverse phytoplankton assemblage throughout the year, with opportunistic species dominating during spring and fall in response to seasonal changes in temperature and nutrients in the southern MAB.

Mid-Atlantic↗
DOE OSTI2025

A universal language for finding mass spectrometry data patterns

Despite being information rich, the vast majority of untargeted mass spectrometry data are underutilized; most analytes are not used for downstream interpretation or reanalysis after publication. The inability to dive into these rich raw mass spectrometry datasets is due to the limited flexibility and scalability of existing software tools. Here, in this study, we introduce a new language, the Mass Spectrometry Query Language (MassQL), and an accompanying software ecosystem that addresses these issues by enabling the community to directly query mass spectrometry data with an expressive set of user-defined mass spectrometry patterns. Illustrated by real-world examples, MassQL provides a data-driven definition of chemical diversity by enabling the reanalysis of all public untargeted metabolomics data, empowering scientists across many disciplines to make new discoveries. MassQL has been widely implemented in multiple open-source and commercial mass spectrometry analysis tools, which enhances the ability, interoperability and reproducibility of mining of mass spectrometry data for the research community.

Damiani, Tito [Czech Academy of Sciences (CAS), Pr↗
DOE OSTI2026

Transfer learning for analysis of collective and non-collective Thomson scattering spectra

Thomson scattering (TS) diagnostics provide reliable, minimally perturbative measurements of fundamental plasma parameters, such as electron density (⁠n e ) and electron temperature (⁠T e ⁠). Deep neural networks can provide accurate estimates of ⁠n e and T e when conventional fitting algorithms may fail, such as when TS spectra are dominated by noise, or when fast analysis is required for real-time operation. Although deep neural networks typically require large training sets, transfer learning can improve model performance on a target task with limited data by leveraging pre-trained models from related source tasks, where select hidden layers are further trained using target data. We present five architecturally diverse deep neural networks, pre-trained on synthetic TS data and adapted for experimentally measured TS data, to evaluate the efficacy of transfer learning in estimating n e and T e in both the collective and non-collective scattering regimes. We evaluate errors in n e and T e estimates as a function of training set size for models trained with and without transfer learning, and we observe decreases in model error from transfer learning when the training set contains ≲ 200 experimentally measured spectra.

Artificial neural networks↗
DOE OSTI2020

Validation of Synthetic U.S. Electric Power Distribution System Data Sets

There is a strong need for synthetic yet realistic distribution system test data sets that are as diverse, large, and complex to solve as real systems. Such data sets can facilitate the development of advanced algorithms and the assessment of emerging distributed energy resources while avoiding the need to acquire proprietary critical infrastructure or private data. Such synthetic data sets, however, are useful only if they are realistic enough to look and behave similarly to actual systems. This paper presents a comprehensive framework for validating synthetic distribution data sets using a three-pronged statistical, operational, and expert validation approach. Furthermore, it also presents a set of statistical and operational metric targets for achieving realistic data sets based on detailed characterization of more than 10,000 real U.S. utility feeders. The paper demonstrates the use of the proposed validation approach to validate three large-scale synthetic data sets developed by the authors representing Santa Fe, New Mexico; Greensboro, North Carolina; and the San Francisco Bay Area, California.

24 POWER TRANSMISSION AND DISTRIBUTION↗