Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data subsampling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
NASA NTRS1994

Vibration (?) spikes during natural rain events

Limited analysis of optical rain gauge (ORG) data from shipboard and ground based sensors has shown the existence of spikes, possibly attributable to sensor vibration, while rain is occurring. An extreme example of this behavior was noted aboard the PRC#5 on the evening of December 24, 1992 as the ship began repositioning during a rain event in the TOGA/COARE IFA. The spikes are readily evident in the one-second resolution data, but may be indistinguishable from natural rain rate fluctuations in subsampled or averaged data. Such spikes result in increased rainfall totals.

Short, David A.↗
DOE OSTI2024

A global gridded dataset for cloud vertical structure from combined CloudSat and CALIPSO observations

Abstract. The vertical structure of clouds has a profound effect on the global energy budget, the global circulation, and the atmospheric hydrological cycle. The CloudSat and Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) missions have taken complementary, colocated observations of cloud vertical structure for over a decade. However, no globally gridded dataset is available to the public for the full length of this unique combined data record. Here we present the 3S-GEOPROF-COMB product (Bertrand et al. 2023, https://doi.org/10.5281/zenodo.8057791), a globally gridded (level 3S) community data product summarizing geometrical profiles (GEOPROF) of hydrometeor occurrence from combined (COMB) CloudSat and CALIPSO data. Our product is calculated from the latest release (R05) of per-orbit (level-2) combined cloud mask profiles. We process a set of cloud cover, vertical cloud fraction, and sampling variables at 2.5, 5, and 10° spatial resolutions and monthly and seasonal temporal resolutions. We address the 2011 reduction in CloudSat data collection with Daylight-Only Operations (DO-Op) mode by subsampling pre-2011 data to mimic DO-Op collection patterns, thereby allowing users to evaluate the impact of the reduced sampling on their analyses. We evaluate our data product against CloudSat-only and CALIPSO-only global-gridded data products as well as four comparable surface-based sites, underscoring the added value of the combined product. Interest in the product is anticipated for the study of cloud processes, cloud–climate interactions, and as a candidate baseline climate data record for comparison to follow-up satellite missions, among other uses.

Bertrand, Leah (ORCID:0009000001607558)↗
NASA NTRS1983

Crop area estimation based on remotely-sensed data with an accurate but costly subsample

Alternatives to sampling-theory stratified and regression estimators of crop production and timber biomass were examined. An alternative estimator which is viewed as especially promising is the errors-in-variable regression estimator. Investigations established the need for caution with this estimator when the ratio of two error variances is not precisely known.

Gunst, R. F.↗
NASA NTRS1985

Crop area estimation based on remotely-sensed data with an accurate but costly subsample

Research activities conducted under the auspices of National Aeronautics and Space Administration Cooperative Agreement NCC 9-9 are discussed. During this contract period research efforts are concentrated in two primary areas. The first are is an investigation of the use of measurement error models as alternatives to least squares regression estimators of crop production or timber biomass. The secondary primary area of investigation is on the estimation of the mixing proportion of two-component mixture models. This report lists publications, technical reports, submitted manuscripts, and oral presentation generated by these research efforts. Possible areas of future research are mentioned.

Gunst, R. F.↗
NASA NTRS2004

The Impact of Subsampling on MODIS Level-3 Statistics of Cloud Optical Thickness and Effective Radius

The MODIS Level-3 optical thickness and effective radius cloud product is a gridded l deg. x 1 deg. dataset that is derived from aggregation and subsampling at 5 km of 1 km, resolution Level-2 orbital swath data (Level-2 granules). This study examines the impact of the 5 km subsampling on the mean, standard deviation and inhomogeneity parameter statistics of optical thickness and effective radius. The methodology is simple and consists of estimating mean errors for a large collection of Terra and Aqua Level-2 granules by taking the difference of the statistics at the original and subsampled resolutions. It is shown that the Level-3 sampling does not affect the various quantities investigated to the same degree, with second order moments suffering greater subsampling errors, as expected. Mean errors drop dramatically when averages over a sufficient number of regions (e.g., monthly and/or latitudinal averages) are taken, pointing to a dominance of errors that are of random nature. When histograms built from subsampled data with the same binning rules as in the Level-3 dataset are used to reconstruct the quantities of interest, the mean errors do not deteriorate significantly. The results in this paper provide guidance to users of MODIS Level-3 optical thickness and effective radius cloud products on the range of errors due to subsampling they should expect and perhaps account for, in scientific work with this dataset. In general, subsampling errors should not be a serious concern when moderate temporal and/or spatial averaging is performed.

Oreopoulos, Lazaros↗
NASA NTRS2016

Study of the Effect of Temporal Sampling Frequency on DSCOVR Observations Using the GEOS-5 Nature Run Results: Cloud Coverage - Part II

This is the second part of a study on how temporal sampling frequency affects satellite retrievals in support of the Deep Space Climate Observatory (DSCOVR) mission. Continuing from Part 1, which looked at Earth's radiation budget, this paper presents the effect of sampling frequency on DSCOVR-derived cloud fraction. The output from NASA's Goddard Earth Observing System version 5 (GEOS-5) Nature Run is used as the "truth". The effect of temporal resolution on potential DSCOVR observations is assessed by subsampling the full Nature Run data. A set of metrics, including uncertainty and absolute error in the subsampled time series, correlation between the original and the subsamples, and Fourier analysis have been used for this study. Results show that, for a given sampling frequency, the uncertainties in the annual mean cloud fraction of the sunlit half of the Earth are larger over land than over ocean. Analysis of correlation coefficients between the subsamples and the original time series demonstrates that even though sampling at certain longer time intervals may not increase the uncertainty in the mean, the subsampled time series is further and further away from the "truth" as the sampling interval becomes larger and larger. Fourier analysis shows that the simulated DSCOVR cloud fraction has underlying periodical features at certain time intervals, such as 8, 12, and 24 h. If the data is subsampled at these frequencies, the uncertainties in the mean cloud fraction are higher. These results provide helpful insights for the DSCOVR temporal sampling strategy.

GEOS-5↗
NASA NTRSDate not supplied

Aircraft Classification Using Radar from Small Unmanned Aerial Systems for Scalable Traffic Management Emergency Response Operations

This work investigates two machine learning techniques: Support Vector Machine (SVM) and Autoencoders (AE)with SVM layer for classification of radar trajectories as General Aviation (GA), fixed-wing small Unmanned Aerial System (sUAS), or not-an-aircraft using radar data recorded from sUAS. Onboard identification of intruder aircraft type is useful for planning avoidance maneuvers and is necessary to provide autonomous systems to meet or exceed the avoidance capability of a human pilot. Aircraft classification can identify intruder aircraft that are not part of the team and may be violating a Temporary Flight Restriction. Aircraft classification is needed in monitoring an airspace where multiple aircraft are teaming on a shared task. Scalable Traffic Management for Emergency Response Operations (STEReO) is a NASA project aimed at improving disaster response by enabling large scale aircraft operations through the teaming of manned aircraft with sUAS to maximize emergency response resources. To this end, this work uses trajectories and radar derived features to classify aircraft from a multirotor sUAS. The AE + SVM generated the strongest classification overall accuracy of 93.5% using the first 4 seconds of radar track data for tracks that activated the avoidance system. Subsampling the available track data increased the available training data with the maximum aircraft recall of 0.94 achieved using the SVM with 1 second track data.

Chester V. Dolph↗
DOE OSTI2020

An investigation of Newton-Sketch and subsampled Newton methods

Sketching, a dimensionality reduction technique, has received much attention in the statistics community. In this paper, we study sketching in the context of Newton's method for solving finite-sum optimization problems in which the number of variables and data points are both large. In this work, we study two forms of sketching that perform dimensionality reduction in data space: Hessian subsampling and randomized Hadamard transformations. Each has its own advantages, and their relative tradeoffs have not been investigated in the optimization literature. Additionally, our study focuses on practical versions of the two methods in which the resulting linear systems of equations are solved approximately, at every iteration, using an iterative solver. The advantages of using the conjugate gradient method vs. a stochastic gradient iteration are revealed through a set of numerical experiments, and a complexity analysis of the Hessian subsampling method is presented.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Performance and calibration of quark/gluon-jet taggers using 140 fb -1 of pp collisions at $\sqrt{s}$ = 13 TeV with the ATLAS detector

The identification of jets originating from quarks and gluons, often referred to as quark/gluon tagging, plays an important role in various analyses performed at the Large Hadron Collider, as Standard Model measurements and searches for new particles decaying to quarks often rely on suppressing a large gluon-induced background. This paper describes the measurement of the efficiencies of quark/gluon taggers developed within the ATLAS Collaboration, using $\sqrt{s}$ = 13 TeV proton–proton collision data with an integrated luminosity of 140 fb -1 collected by the ATLAS experiment. Two taggers with high performances in rejecting jets from gluon over jets from quarks are studied: one tagger is based on requirements on the number of inner-detector tracks associated with the jet, and the other combines several jet substructure observables using a boosted decision tree. A method is established to determine the quark/gluon fraction in data, by using quark/gluon-enriched subsamples defined by the jet pseudorapidity. Differences in tagging efficiency between data and simulation are provided for jets with transverse momentum between 500 GeV and 2 TeV and for multiple tagger working points.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2020

Characterization and differentiation of aluminum powders used in improvised explosive devices – Part 1: Proof of concept of the utility of particle micromorphometry

Abstract Aluminum (Al) powders are commonly used in improvised explosive devices as metallic fuels, a component of explosive mixtures. These powders can be obtained readily from industrial‐scale and consumer products, and produced using unsophisticated “kitchen chemistry” techniques. This research demonstrates the potential of automated particle micromorphometry for comparisons between known source and questioned Al powders recovered from IEDs, as well as for insight into the method of Al powder manufacture. Al powder samples were obtained from legitimate manufacturers, and 56 samples were produced “in‐house” from Al‐containing spray paints and ball‐milled Al foils. Transmitted light microscope images of Al powder particles were acquired using an automated stage with automated z‐focus; 17 size and shape parameters were measured for all particles. Approximately 37,000–2,500,000 particles/sample were analyzed using an open‐source statistical package with customized code. Dimensionality reduction was required for processing the large datasets: eight of the 17 measured variables were selected based on inspection of the correlation matrix. Data from four subsamples from each of the 56 samples produced using “in‐house” methods were analyzed using ANOVA to assess the within‐ and between‐sample variation. High within‐sample variation was noted; however, ANOVA and post‐hoc Tukey's honestly significant difference (HSD) tests demonstrated that the between‐sample variation was substantially larger than the within‐sample variation. Each sample could be differentiated from all other samples in the test set. Future experiments will focus on ways to reduce the within‐sample variation, and additional statistical and microanalytical methods to classify sources and confidently constrain the method of Al powder manufacture.

Baldaino, JenaMarie↗
DOE OSTI2025

Intelligent Sampling of Extreme-Scale Turbulence Datasets for Accurate and Efficient Spatiotemporal Model Training

With the end of Moore’s law and Dennard scaling, efficient training increasingly requires rethinking data volume. Can we train better models with significantly less data via intelligent subsampling? To explore this, we develop SICKLE, a sparse intelligent curation framework for efficient learning, featuring a novel maximum entropy (MaxEnt) sampling approach, scalable training, and energy benchmarking. We compare MaxEnt with random and phase-space sampling on large direct numerical simulation (DNS) datasets of turbulence. Evaluating SICKLE at scale on Frontier, we show that subsampling as a preprocessing step can, in many cases, improve model accuracy and substantially lower energy consumption, with observed reductions of up to 38×.

Brewer, Wes [ORNL] (ORCID:0000000236393956)↗
DOE OSTI2024

Exploration with Scalable Gaussian Process Reinforcement Learning

Exploration is a challenging problem in reinforcement learning (RL), especially in environments with sparse rewards. Quantifying and utilizing the parametric uncertainty has been shown to be paramount for successful exploration [Osband et al., 2018]. Bayesian, or approximately Bayesian, methods present a principled means of estimating the parametric uncertainty in RL problems. Gaussian processes, nonparametric Bayesian models, are often impractical due to poor scalability and computational bottlenecks. We introduce a scalable Gaussian process RL (GPRL) method which directly induces sparsity in the covariance matrix to facilitate faster computation. This is a departure from previous GPRL methods which instead rely on data reduction and subsampling. We compare various covariance-based exploration techniques (Thompson sampling, upper confidence bound, and probabilistic maximum variance) which leverage our scalable GP framework in sparse reward environments. Finally, we show favorable comparison against the bootstrapped deep Q-Network.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Temporal Subsampling Diminishes Small Spatial Scales in Recurrent Neural Network Emulators of Geophysical Turbulence

The immense computational cost of traditional numerical weather and climate models has sparked the development of machine learning (ML) based emulators. Because ML methods benefit from long records of training data, it is common to use data sets that are temporally subsampled relative to the time steps required for the numerical integration of differential equations. Here, we investigate how this often overlooked processing step affects the quality of an emulator's predictions. We implement two ML architectures from a class of methods called reservoir computing: (a) a form of Nonlinear Vector Autoregression (NVAR), and (b) an Echo State Network (ESN). Despite their simplicity, it is well documented that these architectures excel at predicting low dimensional chaotic dynamics. We are therefore motivated to test these architectures in an idealized setting of predicting high dimensional geophysical turbulence as represented by Surface Quasi-Geostrophic dynamics. In all cases, subsampling the training data consistently leads to an increased bias at small spatial scales that resembles numerical diffusion. Interestingly, the NVAR architecture becomes unstable when the temporal resolution is increased, indicating that the polynomial based interactions are insufficient at capturing the detailed nonlinearities of the turbulent flow. The ESN architecture is found to be more robust, suggesting a benefit to the more expensive but more general structure. Spectral errors are reduced by including a penalty on the kinetic energy density spectrum during training, although the subsampling related errors persist. Future work is warranted to understand how the temporal resolution of training data affects other ML architectures.

58 GEOSCIENCES↗
DOE OSTI2025

Variance Preserving Spectral Subsampling

Generating statistically faithful short-duration gamma-ray spectra from a single long measurement is essential in nuclear safeguards, supporting tasks such as algorithm development and machine-learning applications, especially when list-mode data are unavailable. Existing subsampling methods often distort the statistical characteristics of genuine short-duration measurements, leading to biased or unreliable analytical outcomes and thereby undermining downstream tasks. In this work, we compare five subsampling approaches using a benchmark set of 156 genuine replicate spectra collected with a high-purity germanium detector. We evaluate each method with respect to run-to-run variance, channel-to-channel variance, and preservation of total counts (losslessness). Across a wide range of subsampling ratios, only binomial subsampling without replacement consistently reproduces the statistical properties of genuine short-duration spectra, maintaining proper dispersion even in sparse spectral regions and perfectly preserving total counts. These results provide a mathematically principled and practically validated framework for generating synthetically shortened spectra when true short-duration measurements are unavailable.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
DOE OSTI2022

Cosmic ray spectrum of protons plus helium nuclei between 6 and 158 TeV from HAWC data

Here, a measurement with high statistics of the differential energy spectrum of light elements in cosmic rays, in particular, of primary H plus He nuclei, is reported. The spectrum is presented in the energy range from 6 to 158 TeV per nucleus. Data was collected with the High Altitude Water Cherenkov (HAWC) Observatory between June 2015 and June 2019. The analysis was based on a Bayesian unfolding procedure, which was applied on a subsample of vertical HAWC data that was enriched to 82% of events induced by light nuclei. To achieve the mass separation, a cut on the lateral age of air shower data was set guided by predictions of CORSIKA/QGSJET-II-04 simulations. The measured spectrum is consistent with a broken power-law spectrum and shows a kneelike feature at around E = 24.0$^{+3.6}_{-3.1}$ TeV , with a spectral index γ = -2.51 ± 0.02 before the break and with γ = -2.83 ± 0.02 above it. The feature has a statistical significance of 4.1σ. Within systematic uncertainties, the significance of the spectral break is 0.8σ.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Predicting Rare Earth Element Potential in Produced and Geothermal Waters of the United States via Emergent Self-Organizing Maps

This work applies emergent self-organizing map (ESOM) techniques, a form of machine learning, in the multidimensional interpretation and prediction of rare earth element (REE) abundance in produced and geothermal waters in the United States. Visualization of the variables in the ESOM trained using the input data shows that each REE, with the exception of Eu, follows the same distribution patterns and that no single parameter appears to control their distribution. Cross-validation, using a random subsample of the starting data and only using major ions, shows that predictions are generally accurate to within an order of magnitude. Using the same approach, an abridged version of the U.S. Geological Survey Produced Waters Database, Version 2.3 (which includes both data from produced and geothermal waters) was mapped to the ESOM and predicted values were generated for samples that contained enough variables to be effectively mapped. Results show that in general, produced and geothermal waters are predicted to be enriched in REEs by an order of magnitude or more relative to seawater, with maximum predicted enrichments in excess of 1000-fold. Cartographic mapping of the resulting predictions indicates that maximum REE concentrations exceed values in seawater across the majority of geologic basins investigated and that REEs are typically spatially co-associated. The factors causing this co-association were not determined from ESOM analysis, but based on the information currently available, REE content in produced and geothermal waters is not directly controlled by lithology, reservoir temperature, or salinity.

Engle, Mark A. (ORCID:0000000152587374)↗
NASA NTRS2006

Application of the Bootstrap Statistical Method in Deriving Vibroacoustic Specifications

This paper discusses the Bootstrap Method for specification of vibroacoustic test specifications. Vibroacoustic test specifications are necessary to properly accept or qualify a spacecraft and its components for the expected acoustic, random vibration and shock environments seen on an expendable launch vehicle. Traditionally, NASA and the U.S. Air Force have employed methods of Normal Tolerance Limits to derive these test levels based upon the amount of data available, and the probability and confidence levels desired. The Normal Tolerance Limit method contains inherent assumptions about the distribution of the data. The Bootstrap is a distribution-free statistical subsampling method which uses the measured data themselves to establish estimates of statistical measures of random sources. This is achieved through the computation of large numbers of Bootstrap replicates of a data measure of interest and the use of these replicates to derive test levels consistent with the probability and confidence desired. The comparison of the results of these two methods is illustrated via an example utilizing actual spacecraft vibroacoustic data.

Hughes, William O.↗