On the Application of Model-based Parameter Estimation For Mitigating Corrupted Data in Radar Applications.
Abstract not provided.
Maio, Brianna N.↗
SEARCH · Engineering Papers
Results for “data corruption”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
ON THE APPLICATION OF MBPE FOR MITIGATING CORRUPTED DATA IN RADAR APPLICATIONS.
Abstract not provided.
Maio, Brianna N.↗
Characterizing Impacts of Storage Faults on HPC Applications: A methodology and insights
In recent years, the increasing complexity in scientific simulations and emerging demands for training heavy artificial intelligence models require massive and fast data accesses, which urges high-performance computing (HPC) platforms to equip with more advanced storage infrastructures such as solid-state disks (SSDs). While SSDs offer high-performance I/O, it remains unclear about the reliability challenges faced by the HPC applications under the SSD-related failures, in particular, failures resulting in data corruptions. The goal of this paper is to understand the impact of SSD-related data corruptions on the behaviors of complex HPC applications. To this end, we propose FFIS, a FUSE-based fault injection framework that systematically introduces storage faults into the application layer to model the errors originated from SSDs. FFIS is able to plant different I/O related faults into the data returned from underlying file systems, which also enables the investigation on the error resilience characteristics of the scientific file format for the first time. We demonstrate the use of FFIS with three representative real HPC applications, show how each application reacts to the data corruptions, and provide insights on the error resilience of the widely-adopted HDF5 file format for the HPC applications.
Fang, Bo↗
Interpretation of Variations in Modis-Measured Greenness Levels of Amazon Forests During 2000 to 2009
This work investigates variations in satellite-measured greenness of Amazon forests using ten years of NASA Moderate Resolution Imaging Spectroradiometer (MODIS) enhanced vegetation index (EVI) data. Corruption of optical remote sensing data with clouds and aerosols is prevalent in this region; filtering corrupted data causes spatial sampling constraints, as well as reducing the record length, which introduces large biases in estimates of greenness anomalies. The EVI data, analyzed in multiple ways and taking into account EVI accuracy, consistently show a pattern of negligible changes in the greenness levels of forests both in the area affected by drought in 2005 and outside it. Small random patches of anomalous greening and browning-especially prominent in 2009-appear in all ten years, irrespective of contemporaneous variations in precipitation, but with no persistence over time. The fact that over 90% of the EVI anomalies are insignificantly small-within the envelope of error (95% confidence interval) in EVI-warrants cautious interpretation of these results: there were no changes in the greenness of these forests, or if there were changes, the EVI data failed to capture these either because the constituent reflectances were saturated or the moderate resolution precluded viewing small-scale variations. This suggests a need for more accurate and spatially resolved synoptic views from satellite data and corroborating comprehensive ground sampling to understand the greenness dynamics of these forests.
MODIS↗
Method and apparatus for selective and power-aware memory error protection and memory management
A method for providing selective memory error protection responsive to a predictable failure notification associated with at least one portion of a memory in a computing system includes: obtaining an active error correcting code (ECC) configuration corresponding to the portion of the memory; determining whether the active ECC configuration is sufficient to correct at least one error in the portion of the memory affected by the predictable failure notification; when the active ECC configuration is insufficient to correct the error, determining whether data corruption can be tolerated by an application running on the computing system; when data corruption cannot be tolerated by the application, determining whether a stronger ECC level is available and, if a stronger ECC level is available, increasing a strength of the active ECC configuration; and when data corruption can be tolerated, performing page reassignment and aggregation of non-critical data.
97 MATHEMATICS AND COMPUTING↗
Tuning a variational autoencoder for data accountability problem in the Mars Science Laboratory ground data system
The Mars Curiosity rover is frequently sending back engineering and science data that goes through a pipeline of systems before reaching its final destination at the mission operations center making it prone to volume loss and data corruption. A ground data system analysis (GDSA) team is charged with the monitoring of this flow of information and the detection of anomalies in that data in order to request a re-transmission when necessary. This work presents ∆-MADS, a derivative-free optimization method applied for tuning the architecture and hyperparameters of a variational autoencoder trained to detect the data with missing patches in order to assist the GDSA team in their mission.
Lakhmiri, Dounia↗
Comparative Analysis of Soft-Error Detection Strategies: A Case Study with Iterative Methods
Undetected soft errors caused by transient bit flips can lead to silent data corruption (SDC), an undesirable outcome where invalid results pass for valid ones. This has motivated the design of soft error detectors to minimize SDCs. However, the detectors have been studied under different contexts, making comparative evaluation difficult. In this paper, we present the first comprehensive evaluation of four online soft error detection techniques in detecting the adverse impact of soft errors on iterative methods. We observe that, across five iterative methods, the detectors studied achieve high but not perfect detection rates. To understand the potential for improved detection, we evaluate a machine-learning based detector that takes as features that are the runtime features observed by the individual detectors to arrive at their conclusions. Our evaluation demonstrates improved but still far from perfect detection accuracy for the machine learning based detectors. This extensive evaluation demonstrates the need for designing error detectors to handle the evolutionary behavior exhibited by iterative solvers.
Silent data corruption, Soft error detection, Iter↗
MACORD: Online Adaptive Machine Learning Framework for Silent Error Detection
Future HPC systems with ever-increasing resource capacity (such as compute cores, memory and storage) may significantly increase the risks on reliability. Silent data corruptions (SDCs) or silent errors are one of the major sources that corrupt HPC execution results. Unlike fail-stop errors, SDCs are rather harmful and dangerous in that they cannot be detected by hardware. We propose an online machine-learning based silent data corruption detection framework (abbreviated as MACORD) for detecting SDCs in HPC applications. In particular, we comprehensively investigate the prediction ability of a multitude of machine-learning algorithms in our study, and enable the detector to automatically select the best-fit algorithms at runtime to adapt to the data dynamics. Our learning framework exhibits low memory overhead (less than 1%), since it takes only spatial features (i.e., neighboring data values for each data point in the current time step) into the training data. Experiments based on real-world scientific applications/benchmarks show that our framework can get the detection sensitivity (i.e., recall) up to 99% while the false positive rate is limited to 0.1% in most cases, which is one order of magnitude improvement compared with the latest state-of-art spatial technique.
Subasi, Omer↗
“Understanding Robustness Lottery”: A Geometric Visual Comparative Analysis of Neural Network Pruning Approaches
Deep learning approaches have provided state-of-the-art performance in many applications by relying on large and overparameterized neural networks. However, such networks are very brittle and are difficult to deploy on resource-limited platforms. Model pruning, i.e., reducing the size of the network, is a widely adopted strategy that can lead to a more robust and compact model. Many heuristics exist for model pruning, but our understanding of the pruning process remains limited due to the black-box nature of a neural network model. Empirical studies show that some heuristics improve performance whereas others can make models more brittle. Here, this work aims to shed light on how different pruning methods alter the network’s internal feature representation and the corresponding impact on model performance. To facilitate a comprehensive comparison and characterization of the high-dimensional model feature space, we introduce a visual geometric analysis of feature representations. We evaluated a set of critical geometric concepts decomposed from the commonly adopted classification loss and used them to design a visualization system to compare and highlight the impact of pruning on model performance and feature representation. The proposed tool provides an environment for an in-depth comparison of pruning methods and a comprehensive understanding of how the model responds to common data corruption. By leveraging the proposed visualization, machine learning researchers can reveal the similarities between pruning methods and redundancy in robustness evaluation benchmarks, obtain geometric insights about the differences between pruned models that achieve superior robustness performance, and identify samples that are robust or fragile to model pruning and common data corruption.
Li, Zhimin [Univ. of Utah, Salt Lake City, UT (Uni↗
Programmer-guided reliability for extreme-scale applications
We present “programmer-guided reliability” (PGR) as a systematic conceptual approach to address the expected rise in soft errors in coming extreme-scale systems at the application level. The approach involves instrumentation of the application with code to detect data corruption errors. The location and nature of these error detectors are at the discretion of the programmer, who uses their knowledge and experience with the problem domain, the application, the solution algorithms, etc., to determine the most vulnerable areas of the code and the most appropriate ways to detect data corruption. To illustrate the approach, we provide examples of error detectors from four different benchmark-scale applications. We also describe a simple control framework that allows for runtime configuration of the error detectors without recompilation of the application, as well as dynamic reconfiguration during the execution of the application. Lastly, we discuss a number of future directions building on the basic PGR approach, including the incorporation of some general error detectors into the programming environment in order to make them more easily usable by the programmer.
97 MATHEMATICS AND COMPUTING↗
PRESAGE: Protecting Structured Address Generation against Soft Errors
Modern computer scaling trends in pursuit of larger component counts and power efficiency have, unfortunately, lead to less reliable hardware and consequently soft errors escaping into application data ("silent data corruptions"). Techniques to enhance system resilience hinge on the availability of efficient error detectors that have high detection rates, low false positive rates, and lower computational overhead. Unfortunately, efficient detectors to detect faults during address generation (to index large arrays) have not been widely researched. We present a novel lightweight compiler-driven technique called PRESAGE for detecting bit-flips affecting structured address computations. A key insight underlying PRESAGE is that any address computation scheme that flows an already incurred error is better than a scheme that corrupts one particular array access but otherwise (falsely) appears to compute perfectly. Enabling the flow of errors allows one to situate detectors at loop exit points, and helps turn silent corruptions into easily detectable error situations. Our experiments using PolyBench benchmark suite indicate that PRESAGE-based error detectors have a high error-detection rate while incurring low overheads.
Sharma, Vishal C.↗
MO-FG-CAMPUS-IeP2-05: Feasibility Demonstration of High-Voltage Clinical CT and Impact On X-Ray Penetration Through Metal Objects
Purpose: To demonstrate the possibility and quantify the impact of operating a clinical CT scanner at exceptionally high x-ray tube voltage for better penetration through metal objects and facilitating metal artifact reduction. Methods: We categorize metal objects according to the data corruption severeness (level of distortion and complete photon starvation fraction). To demonstrate feasibility and investigate the impact of high voltage scanning we modified a commercial GE LightSpeed VCT scanner (generator and software) to enable CT scans with x-ray tube voltages as high as 175 kVp. A 20 cm diameter water phantom with two metal rods (10 mm stainless and 25 mm titanium) and a water phantom with realistic metal object (spine cage) were used to evaluate the data corruption and image artifacts in the absence of any algorithm correction. We also performed simulations to confirm our understanding of the transmitted photon levels through metal objects with different size and composition. Results: The reconstructed images at 175 kVp still have significant dark shading artifacts, as expected since no special scatter correction or beam hardening was performed but show substantially lower noise and photon starvation than at lower kVp due to better beam penetration. Analysis of the raw data shows that the photon starved data is reduced from over 4% at 140 kVp to below 0.2% at 175 kVp. The simulations indicate that for clinically relevant titanium and stainless objects a 175 kVp tube voltage effectively avoids photon starvation. Conclusion: The use of exceptionally high tube voltage on a clinical CT system is a practical and effective solution to avoid photon starvation caused by certain metal implants. Sparse and hybrid high-voltage protocols are being considered to maintain low patient dose. This opens the door to algorithmic physics-based corrections rather than treating the data as missing and relying on missing data algorithms. Some of the authors are employees of General Electric.
60 APPLIED LIFE SCIENCES↗
Understanding scale-dependent soft-error behavior of scientific applications
Analyzing application fault behavior on large-scale systems is time-consuming and resource-demanding. Currently, researchers need to perform fault injection campaigns at full scale to understand the effects of soft errors on applications and whether these faults result in silent data corruption. Both time and resource requirements greatly limit the scope of the resilience studies that can be currently performed. In this work, we propose a methodology to model application fault behavior at large scale based on a reduced set of experiments performed at small scale. We employ machine learning techniques to accurately model application fault behavior using a set of experiments that can be executed in parallel at small scale. Our methodology drastically reduces the set and the scale of the fault injection experiments to be performed and provides a validated methodology to study application fault behavior at large scale. We show that our methodology can accurately model application fault behavior at large scale by using only small scale experiments. In some cases, we can model the fault behavior of a parallel application running on 4,096 cores with about 90% accuracy based on experiments on a single core.
Kestor, Gokcen G.↗
Impact of duration and missing data on the long-term photovoltaic degradation rate estimation
Accurate quantification of photovoltaic (PV) system degradation rate (R D ) is essential for lifetime yield predictions. Although R D is a critical parameter, its estimation lacks a standardized methodology that can be applied on outdoor field data. The purpose of this paper is to investigate the impact of time period duration and missing data on R D by analyzing the performance of different techniques applied to synthetic PV system data at different linear R D patterns and known noise conditions. The analysis includes the application of different techniques to a 10-year synthetic dataset of a crystalline Silicon PV system, with emulated degradation levels and imputed missing data. Here, the analysis demonstrated that the accuracy of ordinary least squares (OLS), year-on-year (YOY), autoregressive integrated moving average (ARIMA) and robust principal component analysis (RPCA) techniques is affected by the evaluation duration with all techniques converging to lower R D deviations over the 10-year evaluation, apart from RPCA at high degradation levels. Moreover, the estimated R D is strongly affected by the amount of missing data. Filtering out the corrupted data yielded more accurate R D results for all techniques. It is proven that the application of a change-point detection stage is necessary and guidelines for accurate R D estimation are provided.
14 SOLAR ENERGY↗
Redundancy Does Not Imply Fault Tolerance: Analysis of Distributed Storage Reactions to File-System Faults
In this paper, we analyze how modern distributed storage systems behave in the presence of file-system faults such as data corruption and read and write errors. We characterize eight popular distributed storage systems and uncover numerous problems related to file-system fault tolerance. We find that modern distributed systems do not consistently use redundancy to recover from file-system faults: a single file-system fault can cause catastrophic outcomes such as data loss, corruption, and unavailability. We also find that the above outcomes arise due to fundamental problems in file-system fault handling that are common across many systems. Finally, our results have implications for the design of next-generation fault-tolerant distributed and cloud storage systems.
97 MATHEMATICS AND COMPUTING↗
Exploring the interplay of resilience and energy consumption for a task-based partial differential equations preconditioner
Here, we discuss algorithm-based resilience to silent data corruptions (SDCs) in a task-based domain-decomposition preconditioner for partial differential equations (PDEs). The algorithm exploits a reformulation of the PDE as a sampling problem, followed by a solution update through data manipulation that is resilient to SDCs. The implementation is based on a server-client model where all state information is held by the servers, while clients are designed solely as computational units. Scalability tests run up to ~51K cores show a parallel efficiency greater than 90%. We use a 2D elliptic PDE and a fault model based on random single and double bit-flip to demonstrate the resilience of the application to synthetically injected SDC. We discuss two fault scenarios: one based on the corruption of all data of a target task, and the other involving the corruption of a single data point. We show that for our application, given the test problem considered, a four-fold increase in the number of faults only yields a 2% change in the overhead to overcome their presence, from 7% to 9%. We then discuss potential savings in energy consumption via dynamic voltage/frequency scaling, and its interplay with fault-rates, and application overhead.
97 MATHEMATICS AND COMPUTING↗
Cross-geography scientific data transferring trends and behavior
Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.
Liu, Zhengchun↗