SEARCH · Engineering Papers
Results for “Statistical Testing Approach”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Machine learning-enhanced hybrid modeling approach for better identification of a building thermal network model and improved prediction
The gray-box modeling approach, which uses a semi-physical thermal network model, has been widely used in building prediction applications, such as model predictive control (MPC). However, unmeasured disturbances, such as occupants, lighting, and in/exfiltration loads, make it challenging to apply this approach to practical buildings. In this word, we propose a hybrid modeling approach that integrates the gray-box model with a model for unmeasured disturbance. After reviewing several system identification approaches, we systematically designed the unmeasured disturbance model with a model selection process based on statistical tests to make it robust. We generated data based on the building model calibrated by real operational data and then trained the hybrid model for two different weather conditions. The hybrid model approach demonstrates an RMSE reduction of approximately 0.2–0.9 °C and 0.3–2 °C on 1-day ahead temperature prediction compared to the Conventional approach for mild (Berkeley, CA) and cold (Chicago, IL) climates, respectively. In addition, this approach was applied to experimental data obtained from the laboratory building to be used for the MPC application, showing superior prediction performances.
A semblance measure for model comparison
Algorithmic and computational advances have made it possible that geophysical survey and earth model design can be aided by many systematic trial inverse-modelling runs with synthetic data. Such may, for example, come up in machine-learning approaches. Automated image appraisal pertaining to such applications will involve common statistical tests for goodness-of-data fit as a primary evaluation method. However, solution non-uniqueness may render multiple images equivalent in terms of their data fit, requiring secondary categorizers. A logical choice for classifying synthetic-imaging results quantifies the goodness of model fit where a known reference model replaces the observational input. The task of model intercomparison in terms of measuring the resemblance to the reference model poses challenges to common distance-based metrics like root mean square error and mean absolute error. First, distance-based metrics can introduce spurious contributions when smooth models with fuzzy target contours are to be compared against a sharp reference. Second, large differences due to parameter-estimation overshoots can dominate distance metrics. Here, we propose a remedy that is referred to as semblance and is based on the idea of logistic functions, where a binary-dependent variable adds non-zero or zero accumulation terms for the, respectively, passing or failing of preset target thresholds. This classifying approach is amenable to an objective where model feature recognition is primary. Numerical comparisons to distance-based metrics provide evidence for the advantages of the semblance in view of this objective. Geophysical imaging in conjunction with machine-learning is seen as a benefitting upcoming application area.
Comparative Assessment of Data-driven Process Models in Health Information Technology
Process mining for conformance analysis consists of comparing a reference process model against a data-driven process model generated via log files from information technology systems. However, in the absence of a complete reference process model, we found no suggested approaches in the literature to address the need for evaluating process conformance among different healthcare facilities to assess standardization of care. Our goal is to find similarities and dissimilarities in data-driven process models among US Veterans Health Administration (VHA) facilities that can be indicative of patient safety issues. Our hypothesis was that the analysis would not produce statistically significant differences in outcome. We present a unique implementation of conformance analysis in process mining that consists of combining process mining, process mapping and statistical metrics. We illustrate our approach by applying it to the analysis of two clinical radiology order process models generated from healthcare data provided by two similar facilities in the VHA. The comparative assessment showed that about 70% of the orders completed successfully and 30% were not completed due to policy and duplications. Our analysis found a good statistical correlation between both facilities, as the Spearman’s correlation coefficient between facilities for the frequency of cases per total hours was 0.87879, for the frequency of cases by state transition was 0.79702 and for the throughput time per state transition was 0.63582. Additional statistical analyses using the Mann-Whitney U test and the root mean square error both produced values that were not significant. The foregoing approach validated our hypothesis by demonstrating a good statistical correlation of data describing the flow of clinical radiology orders absent a credible reference model. Finding good agreement between both facilities was important in confirming that the clinical orders flow in a similar manner, suggesting standardization of care.
Functional vs. Traditional Analysis in Biomechanical Gait Data: An Alternative Statistical Approach
In human motion studies, discrete points such as peak or average kinematic values are commonly selected to test hypotheses. The purpose of this study was to describe a functional data analysis and describe the advantages of using functional data analyses when compared with a traditional analysis of variance (ANOVA) approach. Nineteen healthy participants (age: 22 ± 2 yrs, body height: 1.7 ± 0.1 m, body mass: 73 ± 16 kg) walked under two different conditions: control and pain+effusion. Pain+effusion was induced by injection of sterile saline into the joint capsule and hypertonic saline into the infrapatellar fat pad. Sagittal-plane ankle, knee, and hip joint kinematics were recorded and compared following injections using 2×2 mixed model ANOVAs and FANOVAs. The results of ANOVAs detected a condition × time interaction for the peak ankle (F1,18 = 8.56, p = 0.01) and hip joint angle (F1,18 = 5.77, p = 0.03), but did not for the knee joint angle (F1,18 = 0.36, p = 0.56). The functional data analysis, however, found several differences at initial contact (ankle and knee joint), in the mid-stance (each joint) and at toe off (ankle). Although a traditional ANOVA is often appropriate for discrete or summary data, in biomechanical applications, the functional data analysis could be a beneficial alternative. Thus when using the functional data analysis approach, a researcher can (1) evaluate the entire data as a function, and (2) detect the location and magnitude of differences within the evaluated function.
FolpsD: combining EFT and phenomenological approaches for joint power spectrum and bispectrum analyses
We present a theoretical model for the power spectrum and bispectrum of galaxy clustering that exploits the complementarity between small-scale power spectrum information and large-scale bispectrum measurements. We extend the FOLPS code by combining its one-loop EFT galaxy power spectrum with a tree-level galaxy bispectrum projected onto the tripolar spherical harmonics (Sugiyama) basis. To access additional small-scale information, we also consider a line-of-sight damping factor in both statistics, mirroring approaches commonly used in studies of redshift-space distortions. We test the model using DESI DR2 galaxy mocks. Even without damping, the joint analysis of the EFT power spectrum and bispectrum significantly improves constraints and reduces parameter degeneracies relative to power spectrum analyses alone. For LRG-like samples, including the damping further extends the range beyond $k\sim 0.3 \,h \text{Mpc}^{-1}$ in the power spectrum and $k \sim 0.24 \,h \text{Mpc}^{-1}$ in the bispectrum without introducing statistically significant parameter biases. This leads to up to $\sim 30\%$ tighter constraints on $A_s$ and $ω_{cdm}$. For low signal-to-noise tracers such as QSOs, however, the damping parameters are weakly constrained and can absorb noise fluctuations, leading to shifts in inferred parameters. Similar limitations may arise in models where cosmological information is encoded in power-spectrum shape features degenerate with the damping, such as scenarios with massive neutrinos. In contrast, for $w_0w_a$CDM we obtain $15\%$ and $21\%$ tighter constraints on $w_0$ and $w_a$, respectively, yielding a deviation from constant dark energy at slightly more than the $1σ$ level using full-shape information alone. The code is publicly available at https://github.com/cosmodesi/FolpsD
SU-C-BRD-01: A Statistical Modeling Method for Quality Control of Intensity- Modulated Radiation Therapy Planning
Purpose: To apply a statistical modeling approach, threshold modeling (TM), for quality control of intensity-modulated radiation therapy (IMRT) treatment plans. Methods: A quantitative measure, which was the weighted sum of violations of dose/dose-volume constraints, was first developed to represent the quality of each IMRT plan. Threshold modeling approach, which is is an extension of extreme value theory in statistics and is an effect way to model extreme values, was then applied to analyze the quality of the plans summarized by our quantitative measures. Our approach modeled the plans generated by planners as a series of independent and identically distributed random variables and described the behaviors of them if the plan quality was controlled below certain threshold. We tested our approach with five locally advanced head and neck cancer patients retrospectively. Two statistics were incorporated for numerical analysis: probability of quality improvement (PQI) of the plans and expected amount of improvement on the quantitative measure (EQI). Results: After clinical planners generated 15 plans for each patient, we applied our approach to obtain the PQI and EQI as if planners would generate additional 15 plans. For two of the patients, the PQI was significantly higher than the other three (0.17 and 0.18 comparing to 0.08, 0.01 and 0.01). The actual percentage of the additional 15 plans that outperformed the best of initial 15 plans was 20% and 27% comparing to 11%, 0% and 0%. EQI for the two potential patients were 34.5 and 32.9 and the rest of three patients were 9.9, 1.4 and 6.6. The actual improvements obtained were 28.3 and 20.5 comparing to 6.2, 0 and 0. Conclusion: TM is capable of reliably identifying the potential quality improvement of IMRT plans. It provides clinicians an effective tool to assess the trade-off between extra planning effort and achievable plan quality. This work was supported in part by NIH/NCI grant CA130814.
Report on Hydrogen Content Measurements of Yttrium Hydrides
This report describes the hydrogen content measurements of yttrium samples as irradiated at Advanced Test Reactor as part of the Microreactor Program. Irradiated samples were prepared at the facilities of Analytical Research Laboratory (ARL) of Materials and Fuels Complex (MFC). Hydrogen content measurements were performed on reduced size specimens using an inert gas fusion analyzer. For a single sample, replicate samples were prepared and tested to improve statistics of results. Uncertainty analyses were conducted using two approaches to determine the hydrogen content variations in the samples. Hydrogen content of irradiated specimens were both lower and higher than the expected values, indicating stoichiometry variations. Hydrogen was detected in all samples, even for the cracked capsule’s specimens. Results also suggested a potential hydrogen redistribution is present between samples inside the capsules.
Preventing Failures By Dataset Shift Detection in Safety-Critical Graph Applications
Dataset shift refers to the problem where the input data distribution may change over time (e.g., between training and test stages). Since this can be a critical bottleneck in several safety-critical applications such as healthcare, drug-discovery, etc., dataset shift detection has become an important research issue in machine learning. Though several existing efforts have focused on image/video data, applications with graph-structured data have not received sufficient attention. Therefore, in this paper, we investigate the problem of detecting shifts in graph structured data through the lens of statistical hypothesis testing. Specifically, we propose a practical two-sample test based approach for shift detection in large-scale graph structured data. Our approach is very flexible in that it is suitable for both undirected and directed graphs, and eliminates the need for equal sample sizes. Using empirical studies, we demonstrate the effectiveness of the proposed test in detecting dataset shifts. We also corroborate these findings using real-world datasets, characterized by directed graphs and a large number of nodes.
Modeling the Functional Forms of Grid Disturbances
This report introduces a functional form that may be used to quantitatively predict the impacts of new grid tools and changing system qualities on the likelihoods, durations, and depths of various grid disturbances. Each disturbance scenario is modeled to have three component stages—avoidance, reactance and recovery, which together parametrically estimate one disturbance’s impacts. The modeled scenario is then placed and replicated within an analysis period to represent the likelihood or frequency of the scenario and its consequent impacts. Whereas analysts have struggled to define and apply metrics for grid resilience, the functional form introduced by this report shares units of measurement with accepted grid-status measures (e.g., numbers of customers currently experiencing a service outage). Furthermore, the integrated and averaged functional form over an analysis period provides a meaningful normalized performance metric (e.g., customer outage minutes per year) that is ultimately independent of the duration of the period of. The approach may be applied similarly regardless of the severity or frequency of the disturbances that are being analyzed. Because metrics can be chosen to be identical in both the hypothetical future and the actual historical past, the historical past eventually becomes the test of the future predictions, at least in a statistical sense. The authors originally developed this approach to facilitate analysis of the effects of transactive energy (TE) systems effects on electric power grid resilience. TE systems invite energy suppliers and consumers to actively collaborate toward the discovery of, and their responses to, the locational value of energy. The findings from this process are often embodied as energy prices, the dynamics of which indicate the locational value of energy and can further represent important grid service needs. While some academic papers claim to quantify the value of a specific TE system design toward grid resilience, the answer, in general, has been elusive. Not only do multiple and conflicting definitions of resilience and reliability exist, but countless TE systems are being invented. We conclude the following: (1) The ideal analysis should harmonize rather than differentiate resilience and reliability. Therefore, this report uses the more general term disturbance whenever the overloaded terms resilience and reliability can be avoided. (2) The effectiveness of TE systems must be mapped to underlying qualities of a TE system, thereby avoiding presumptions that every TE design offers similar advantages. The authors seek to evaluate the parametric effects of TE system qualities (e.g., spatial granularity, granularity of time steps, length of future prediction horizon) on avoiding, reacting to, and recovering from grid disturbances. Furthermore, any advantages (or disadvantages) must be fairly compared with the many alternative tools, systems, and strategies that might offer comparable benefits.
Fourier transform ion cyclotron resonance mass spectrometry (FT-ICR-MS) peak intensity normalization for complex mixture analyses
Fourier transform ion cyclotron resonance mass spectrometry (FT-ICR-MS) is a preferred technique for analyzing complex organic mixtures. Currently, there is no consensus normalization approach, nor an objective method for selecting one, for quantitative analyses of FT-ICR-MS data. We investigate a method to evaluate and score the amount of bias various normalization approaches introduce into the data. We evaluate the ability of the Statistical Procedure for the Analysis of Normalization Strategies (SPANS) to guide selection of appropriate normalization approaches for two different FT-ICR-MS datasets. Further, we test the robustness of SPANS results to changes in SPANS parameter values and assess the impact of using various normalization approaches on downstream statistical analyses. The normalization approach identified by SPANS differed for the two datasets. Normalization methods impacted the statistical significance of peaks differently, underscoring the importance of carefully evaluating potential methods. More consistent SPANS scores resulted when at least 120 significant peaks are used, where larger sets of peaks were obtained by increasing the p-value threshold. Interestingly, we show that Total Sum Scaling and highest peak normalization, used in previous studies, underperformed relative to SPANS-recommended normalization approaches. While there is no single, best normalization method for all datasets, SPANS provides a mechanism to identify an appropriate normalization method for analyzing FT-ICR-MS data quantitatively. As a result, the number of peaks used in the background distributions of SPANS contributes more significantly to the reproducibility of results than the p-value thresholds used to obtain those peaks.
Multidimensional scaling informed by F -statistic: Visualizing grouped microbiome data with inference
Multidimensional scaling (MDS) is a widely used dimensionality reduction technique in microbial ecology data analysis that captures the multivariate structure of the data while preserving pairwise distances between samples. While improvements in MDS have enhanced the ability to reveal group-specific data patterns, these MDS-based methods require prior assumptions for inference, limiting their application in general microbiome analysis. Here, in this study, we introduce a new MDS-based ordination method, “F-informed MDS,” which configures the data distribution based on the F-statistic, the ratio of dispersion between groups sharing common and different characteristics. Using semisynthetic datasets, we demonstrate that the proposed method is robust to hyperparameter selection while maintaining statistical significance throughout the ordination process. Various quality metrics for evaluating dimensionality reduction confirm that F-informed MDS is comparable to state-of-the-art methods in preserving both local and global data structures. Its application to a diatom-associated bacterial community suggests the role of this new method in interpreting the community’s response to the host. Our approach offers a well-founded refinement of MDS that aligns with statistical test results, which can be beneficial for broader multidimensional data analyses in microbiology and ecology. This new visualization tool can be incorporated into standard microbiome data analyses.
To have value, comparisons of high-throughput phenotyping methods need statistical tests of bias and variance
The gap between genomics and phenomics is narrowing. The rate at which it is narrowing, however, is being slowed by improper statistical comparison of methods. Quantification using Pearson’s correlation coefficient ( r ) is commonly used to assess method quality, but it is an often misleading statistic for this purpose as it is unable to provide information about the relative quality of two methods. Using r can both erroneously discount methods that are inherently more precise and validate methods that are less accurate. These errors occur because of logical flaws inherent in the use of r when comparing methods, not as a problem of limited sample size or the unavoidable possibility of a type I error. A popular alternative to using r is to measure the limits of agreement (LOA). However both r and LOA fail to identify which instrument is more or less variable than the other and can lead to incorrect conclusions about method quality. An alternative approach, comparing variances of methods, requires repeated measurements of the same subject, but avoids incorrect conclusions. Variance comparison is arguably the most important component of method validation and, thus, when repeated measurements are possible, variance comparison provides considerable value to these studies. Statistical tests to compare variances presented here are well established, easy to interpret and ubiquitously available. The widespread use of r has potentially led to numerous incorrect conclusions about method quality, hampering development, and the approach described here would be useful to advance high throughput phenotyping methods but can also extend into any branch of science. The adoption of the statistical techniques outlined in this paper will help speed the adoption of new high throughput phenotyping techniques by indicating when one should reject a new method, outright replace an old method or conditionally use a new method.
Reconstruction of a direction-dependent primordial power spectrum from Planck CMB data
We consider the possibility that the primordial curvature perturbation is direction-dependent. To first order this is parameterised by a quadrupolar modulation of the power spectrum and results in statistical anisotropy of the CMB, which can be quantified using `bipolar spherical harmonics'. We compute these for the Planck DR2-2015 SMICA map and estimate the noise covariance from Planck Full Focal Plane 9 simulations. A constant quadrupolar modulation is detected with 2.2$σ$ significance, dropping to 2$σ$ when the primordial power is assumed to scale with wave number k as a power law. Going beyond previous work we now allow the spectrum to have arbitrary scale-dependence. Our non-parametric reconstruction then suggests several spectral features, the most prominent at k ~ 0.006 Mpc -1 . When a constant quadrupolar modulation is fitted to data in the range 0.005 ≤ k/Mpc -1 ≤ 0.008, its preferred directions are found to be related to the cosmic hemispherical asymmetry and the CMB dipole. To determine the significance we apply two test statistics to our reconstructions of the quadrupolar modulation from data, against reconstructions of realisations of noise only. With a test statistic sensitive only to the amplitude of the modulation, the reconstructions from the multipole range 30 ≤ ℓ ≤ 1200 are unusual with 2.1$σ$ significance. With the second test statistic, sensitive also to the direction, the significance rises to 6.9$σ$. Our approach is easily generalised to include other data sets such as polarisation, large-scale structure and forthcoming 21-cm line observations which will enable these anomalies to be investigated further.
Life Prediction of Large Lithium-Ion Battery Packs with Active and Passive Balancing
Lithium-ion battery packs take a major part of large-scale stationary energy storage systems. One challenge in reducing battery pack cost is to reduce pack size without compromising pack service performance and lifespan. Prognostic life model can be a powerful tool to handle the state of health (SOH) estimate and enable active life balancing strategy to reduce cell imbalance and extend pack life. This work proposed a life model using both empirical and physical-based approaches. The life model described the compounding effect of different degradations on the entire cell with an empirical model. Then its lower-level submodels considered the complex physical links between testing statistics (state of charge level, C-rate level, duty cycles, etc.) and the degradation reaction rates with respect to specific aging mechanisms. The hybrid approach made the life model generic, robust and stable regardless of battery chemistry and application usage. The model was validated with a custom pack with both passive and active balancing systems implemented, which created four different aging paths in the pack. The life model successfully captured the aging trajectories of all four paths. The life model prediction errors on capacity fade and resistance growth were within +/-3% and +/-5% of the experiment measurements.
Bayesian stability and force modeling for uncertain machining processes
Accurately simulating machining operations requires knowledge of the cutting force model and system frequency response. However, this data is collected using specialized instruments in an ex-situ manner. Bayesian statistical methods instead learn the system parameters using cutting test data, but to date, these approaches have only considered milling stability. This paper presents a physics-based Bayesian framework which incorporates both spindle power and milling stability. Initial probabilistic descriptions of the system parameters are propagated through a set of physics functions to form probabilistic predictions about the milling process. The system parameters are then updated using automatically selected cutting tests to reduce parameter uncertainty and identify more productive cutting conditions, where spindle power measurements are used to learn the cutting force model. The framework is demonstrated through both numerical and experimental case studies. Results show that the approach accurately identifies both the system natural frequency and cutting force model.
Detecting differential protein abundance by combining peptide level P -values
The majority of methods for detecting differentially abundant proteins between samples in label-free LC-MS bottom-up proteomics experiments rely on statistically testing inferred protein abundances derived from peptide ionization intensities or averaging peptide level statistics. Here in this paper, we statistically test peptide ionization intensities directly and combine the resulting dependent P-values using the Empirical Brown's Method (EBM), avoiding error introduced through the estimation of protein abundances or summarizing test statistics. We show that on a spike-in proteomics dataset, a peptide level approach using EBM outperforms differential abundance detection using a protein level approach and several analysis workflows, including MSstats. Additionally, we demonstrate the effectiveness of this approach by detecting enriched proteins from an activity-based protein profiling dataset.
Retrofit California: Better Buildings Program Overview
Energy efficiency retrofits (also called upgrades) are widely recognized as a critical component to achieving energy savings in the building sector to help lower greenhouse gas (GHG) emissions. To date, however, upgrades have accounted for only a small percentage of aggregate energy savings in building stock, both in California and nationally. Although the measures and technologies to retrofit a building to become energy efficient are readily deployed, establishing this model as a standard practice remains elusive. Retrofit California sought to develop and test new program models to increase participation in the energy upgrade market in California. The Program encompassed 24 pilot projects, conducted between 2010 and mid-2013 and funded through a $30 million American Recovery and Reinvestment Act (ARRA) grant from the U.S. Department of Energy’s (DOE) Better Buildings Neighborhood Program (BBNP). The broad scope of the Program can be seen in the involvement of the following regionally based Grant Partners: Los Angeles County (as prime grantee); Association of Bay Area Governments (ABAG), consisting of: StopWaste.org for Alameda County, Regional Climate Protection Authority (RCPA) for Sonoma County, SF Environment for the City and County of San Francisco, and City of San Jose; California Center for Sustainable Energy (CCSE) for the San Diego region; Sacramento Municipal Utilities District (SMUD). Within these jurisdictions, nine different types of pilots were tested with the common goal of identifying, informing, and educating the people most likely to undertake energy upgrades (both homeowners and contractors), and to provide them with incentives and resources to facilitate the process. Despite its limited duration, Retrofit California undoubtedly succeeded in increasing awareness and education among home and property owners, as well as contractors, realtors, and community leaders. However, program results indicate that a longer timeframe will be needed to transform the market and establish energy retrofits as the new paradigm. Innovations such as Flex Path, which came about because of barriers encountered during the Program, have already shown promise and are enabling increased participation. Together, the pilots represent an unprecedented effort to identify and address market barriers to energy efficiency upgrades and to provide lessons learned to shape future program planning and implementation. The statistics reflects the scope of the marketing and outreach campaigns, which tested a variety of approaches to increase understanding of the benefits of energy upgrades to drive participation in the Program. More traditional methods such as TV and radio advertisements were complimented by innovative community based social marketing campaigns that sought to leverage the trusted status of neighborhood organizations and leaders in order to motivate their constituents to undertake retrofits. The remainder of this report provides an overview of Retrofit California including brief summaries of the pilots’ main components and highlights, followed by the major findings or takeaway lessons from the approaches that were tested. Eleven of the pilots will be continued, with modifications, under the ratepayer-funded Regional Energy Networks. Involvement in the RENS by many of the Retrofit California partners will ensure that early lessons learned are carried forward to guide future programs for energy upgrades in California.