Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data corruption”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Power System Event Detection Using the Energy Detector: A Performance Analysis

As the grid becomes smarter, the need to accurately detect, predict, and classify waveform phenomena is growing. When it comes to detecting high-frequency behaviors, i.e. transients, it is especially important to employ an event detection system that is able to accurately uncover these types of disturbances that would otherwise be lost with traditional hardware. In this paper, we first present the energy detector; a waveform event detection system that adaptively monitors a signal's energy and picks out high-frequency events that deviate from the nominal state. Secondly, we evaluate the performance of this detector against waveform data that have been corrupted by sensor irregularities. Using Oak Ridge National Laboratory's sensor testbed, we are able to show the results of the detector's performance against events that have been corrupted by three distinct sensor types, and examine how these results change with multiple trials. The results show excellent performance when detecting the beginning of an anomalous event with an average of less than 1% error.

Ekti, Ali Riza↗
DOE OSTI2024

Complete and Correct Transfer of Information (CACTI)

Many distributed systems, file transfer mechanisms, and message passing systems offer reliability mechanisms such as acknowledgements, retries, and durability. While these tools may be “good enough” for their typical use cases, they may not offer sufficient coverage for the wide range of faults that impact data transfers and communication. A gap in the reliability measures may lead to some small amount of data loss. Some high-consequence systems cannot tolerate the loss or corruption of even a single record. We present seven principles that will counter a wide range of faults and protect against data loss and corruption. These principles bring together lessons learned from a wide range of technologies and can inform appropriate system design and application usage. These principles will help readers reason on how prevent data loss in a multi-hop pipeline and how to properly use tools that may have a deficiency in reliability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Community Energy by Design: A Simulation-Based Design Workflow Using Measured Data Clustering to Calibrate Urban Building Energy Models (UBEMs)

This paper presents a workflow that informs urban design decisions using measured data clustering to calibrate urban building energy models. The method's goal is to support urban design in terms of form, building systems configurations, as well as influencing user behavior aspects in the built environment through a systemic analysis of measured data to develop reliable future-case design scenario energy models. Detailed data on appliance-level electricity use are employed via data clustering to calibrate a urban building energy model for the Mueller community in Austin, TX, USA. The data were collected by the Pecan Street Institute for a year in 2014 from consumers in Austin and other surrounding cities. First, collected energy data were restructured and cleaned from corrupt and/or missing information. Second, in order to identify common energy use patterns, a model-based clustering algorithm for functional data was applied. Behavioral/usage profiles are determined through clustering and translated into usage schedules and behaviors. As a result, an urban building energy model built in the urban modeling interface (umi) was calibrated, with fully calibrated and semi-calibrated buildings, within a maximum calibration error margin of 14% for daily-scale averages. Finally, an illustration of calibrated-urban building energy model design case scenarios is presented, and implications on community energy potential effects are discussed.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

Boosting the signal-to-noise of low-field MRI with deep learning image reconstruction

Recent years have seen a resurgence of interest in inexpensive low magnetic field (< 0.3 T) MRI systems mainly due to advances in magnet, coil and gradient set designs. Most of these advances have focused on improving hardware and signal acquisition strategies, and far less on the use of advanced image reconstruction methods to improve attainable image quality at low field. We describe here the use of our end-to-end deep neural network approach (AUTOMAP) to improve the image quality of highly noise-corrupted low-field MRI data. We compare the performance of this approach to two additional state-of-the-art denoising pipelines. We find that AUTOMAP improves image reconstruction of data acquired on two very different low-field MRI systems: human brain data acquired at 6.5 mT, and plant root data acquired at 47 mT, demonstrating SNR gains above Fourier reconstruction by factors of 1.5- to 4.5-fold, and 3-fold, respectively. In these applications, AUTOMAP outperformed two different contemporary image-based denoising algorithms, and suppressed noise-like spike artifacts in the reconstructed images. The impact of domain-specific training corpora on the reconstruction performance is discussed. The AUTOMAP approach to image reconstruction will enable significant image quality improvements at low-field, especially in highly noise-corrupted environments.

42 ENGINEERING↗
DOE OSTI2014

Fault-tolerant dynamic task graph scheduling

In this paper, we present an approach to fault tolerant execution of dynamic task graphs scheduled using work stealing. In particular, we focus on selective and localized recovery of tasks in the presence of soft faults. We elicit from the user the basic task graph structure in terms of successor and predecessor relationships. The work stealing-based algorithm to schedule such a task graph is augmented to enable recovery when the data and meta-data associated with a task get corrupted. We use this redundancy, and the knowledge of the task graph structure, to selectively recover from faults with low space and time overheads. We show that the fault tolerant design retains the essential properties of the underlying work stealing-based task scheduling algorithm, and that the fault tolerant execution is asymptotically optimal when task re-execution is taken into account. Experimental evaluation demonstrates the low cost of recovery under various fault scenarios.

fault tolerance↗
DOE OSTI2016

Makalu: fast recoverable allocation of non-volatile memory

Byte addressable non-volatile memory (NVRAM) is likely to supplement, and perhaps eventually replace, DRAM. Applications can then persist data structures directly in memory instead of serializing them and storing them onto a durable block device. However, failures during execution can leave data structures in NVRAM unreachable or corrupt. In this paper, we present Makalu, a system that addresses non-volatile memory management. Makalu offers an integrated allocator and recovery-time garbage collector that maintains internal consistency, avoids NVRAM memory leaks, and is efficient, all in the face of failures. We show that a careful allocator design can support a less restrictive and a much more familiar programming model than existing persistent memory allocators. Our allocator significantly reduces the per allocation persistence overhead by lazily persisting non-essential metadata and by employing a post-failure recovery-time garbage collector. Experimental results show that the resulting online speed and scalability of our allocator are comparable to well-known transient allocators, and significantly better than state-of-the-art persistent allocators.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2013

Analysis of signals under compositional noise with applications to SONAR data

In this paper, we consider the problem of denoising and classification of SONAR signals observed under compositional noise, i.e., they have been warped randomly along the x-axis. The traditional techniques do not account for such noise and, consequently, cannot provide a robust classification of signals. We apply a recent framework that: 1) uses a distance-based objective function for data alignment and noise reduction; and 2) leads to warping-invariant distances between signals for robust clustering and classification. We use this framework to introduce two distances that can be used for signal classification: a) a y-distance, which is the distance between the aligned signals; and b) an x-distance that measures the amount of warping needed to align the signals. We focus on the task of clustering and classifying objects, using acoustic spectrum (acoustic color), which is complicated by the uncertainties in aspect angles at data collections. Small changes in the aspect angles corrupt signals in a way that amounts to compositional noise. As a result, we demonstrate the use of the developed metrics in classification of acoustic color data and highlight improvements in signal classification over current methods.

47 OTHER INSTRUMENTATION↗
DOE OSTI2015

Improving the trust in results of numerical simulations and scientific data analytics

This white paper investigates several key aspects of the trust that a user can give to the results of numerical simulations and scientific data analytics. In this document, the notion of trust is related to the integrity of numerical simulations and data analytics applications. This white paper complements the DOE ASCR report on Cybersecurity for Scientific Computing Integrity by (1) exploring the sources of trust loss; (2) reviewing the definitions of trust in several areas; (3) providing numerous cases of result alteration, some of them leading to catastrophic failures; (4) examining the current notion of trust in numerical simulation and scientific data analytics; (5) providing a gap analysis; and (6) suggesting two important research directions and their respective research topics. To simplify the presentation without loss of generality, we consider that trust in results can be lost (or the results’ integrity impaired) because of any form of corruption happening during the execution of the numerical simulation or the data analytics application. In general, the sources of such corruption are threefold: errors, bugs, and attacks. Current applications are already using techniques to deal with different types of corruption. However, not all potential corruptions are covered by these techniques. We firmly believe that the current level of trust that a user has in the results is at least partially founded on ignorance of this issue or the hope that no undetected corruptions will occur during the execution. This white paper explores the notion of trust and suggests recommendations for developing a more scientifically grounded notion of trust in numerical simulation and scientific data analytics. We first formulate the problem and show that it goes beyond previous questions regarding the quality of results such as V&V, uncertainly quantification, and data assimilation. We then explore the complexity of this difficult problem, and we sketch complementary general approaches to address it. This paper does not focus on the trust that the execution will actually complete. The product of simulation or of data analytic executions is the final element of a potentially long chain of transformations, where each stage has the potential to introduce harmful corruptions. These corruptions may produce results that deviate from the user-expected accuracy without notifying the user of this deviation. There are many potential sources of corruption before and during the execution; consequently, in this white paper we do not focus on the protection of the end result after the execution.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Exploiting data representation for fault tolerance

Incorrect computer hardware behavior may corrupt intermediate computations in numerical algorithms, possibly resulting in incorrect answers. Prior work models misbehaving hardware by randomly flipping bits in memory. We start by accepting this premise, and present an analytic model for the error introduced by a bit flip in an IEEE 754 floating-point number. We then relate this finding to the linear algebra concepts of normalization and matrix equilibration. In particular, we present a case study illustrating that normalizing both vector inputs of a dot product minimizes the probability of a single bit flip causing a large error in the dot product's result. Moreover, the absolute error is either less than one or very large, which allows detection of large errors. Then, we apply this to the GMRES iterative solver. We count all possible errors that can be introduced through faults in arithmetic in the computationally intensive orthogonalization phase of GMRES, and show that when the matrix is equilibrated, the absolute error is bounded above by one.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning

In overhead image segmentation tasks, including additional spectral bands beyond the traditional RGB channels can improve model performance. However, it is still unclear how incorporating this additional data impacts model robustness to adversarial attacks and natural perturbations. For adversarial robustness, the additional in-formation could improve the model’s ability to distinguish malicious inputs, or simply provide new attack avenues and vulnerabilities. For natural perturbations, the additional information could better inform model decisions and weaken perturbation effects or have no significant influence at all. In this work, we seek to characterize the performance and robustness of a multispectral (RGB and near infrared) image segmentation model subjected to adversarial attacks and natural perturbations. While existing adversarial and natural robustness research has focused primarily on digital perturbations, we prioritize on creating realistic perturbations designed with physical world conditions in mind. For adversarial robustness, we focus on data poisoning attacks whereas for natural robustness, we focus on extending ImageNet-C common corruptions for fog and snow that coherently and self-consistently perturbs the input data. Overall, we find both RGB and multispectral models are vulnerable to data poisoning attacks regardless of input or fusion architectures and that while physically-realizable natural perturbations still degrade model performance, the impact differs based on fusion architecture and input data.

Deep learning, multispectral images, multimodal fu↗
DOE OSTI2023

PROTEUS: Machine Learning Driven Resilience for Extreme-scale Systems

The objective of this project is to design, develop, and evaluate scalable software to enhance resilience, data checkpointing, program restart, and analysis. The proposed tasks are to 1) develop scalable machine learning techniques to learn temporal change patterns in a scalable and in-situ manner, and to minimize data movement and maximize learning locally closest to data; 2) design a concise data representation and indexing mechanism to capture the distribution of changes in data that can guarantee point-wise user-defined tolerable errors while reducing the data storage requirements by an order of magnitude or more; 3) develop data reduction techniques as library modules; 4) exploit local SSD for minimizing data movement in storage hierarchy; 5) develop anomaly detection algorithms that can predict corruptions based on learning of emerging patterns; 6) develop software libraries to be incorporated within widely used data formats and APIs; and 7) evaluate the proposed software using DOE scientific applications. The outcomes of the proposed work are to satisfy many synergistic data reduction and resilience requirements for large-scale data intensive applications executed on extreme-scale computing systems. The developed mechanism for error-bound data approximation is directly applicable to existing scientific applications. Through machine learning from historical events and change distribution, this work will enable anomaly detection for DOE computer facility.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking

In the past few years, Generative Adversarial Networks (GANs) have dramatically advanced our ability to represent and parameterize high-dimensional, non-linear image manifolds. As a result, they have been widely adopted across a variety of applications, ranging from challenging inverse problems like image completion, to problems such as anomaly detection and adversarial defense. A recurring theme in many of these applications is the notion of projecting an image observation onto the manifold that is inferred by the generator. In this context, Projected Gradient Descent (PGD) has been the most popular approach, which essentially optimizes for a latent vector that minimizes the discrepancy between a generated image and the given observation. However, PGD is a brittle optimization technique that fails to identify the right projection (or latent vector) when the observation is corrupted, or perturbed even by a small amount. Such corruptions are common in the real world, for example images in the wild come with unknown crops, rotations, missing pixels, or other kinds of non-linear distributional shifts which break current encoding methods, rendering downstream applications unusable. To address this, we propose corruption mimicking—a new robust projection technique, that utilizes a surrogate network to approximate the unknown corruption directly at test time, without the need for additional supervision or data augmentation. The proposed method is significantly more robust than PGD and other competing methods under a wide variety of corruptions, thereby enabling a more effective use of GANs in real-world applications. Finally, more importantly, we show that our approach produces state-of-the-art performance in several GAN-based applications—anomaly detection, domain adaptation, and adversarial defense, that benefit from an accurate projection.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Asi Nuclear Energy Sensors Data Portal Chatbot And Data Structuring Tool

The Idaho National Laboratory (INL) is advancing the development of an AI-powered chatbot and data structuring tool specifically designed to accelerate data mining processes for sensor-related information and seamlessly integrate the results into the ASI Sensors Data Portal (https://nes.energy.gov/). By doing so, the software aims to enhance the accessibility, usability, and organization of sensor data for nuclear energy applications. The software initial phase focuses on retrieving comprehensive datasets, prioritizing the past five years of publicly available information from the Office of Scientific and Technical Information (OSTI). These datasets will be meticulously processed to ensure compatibility, employing cleaning and preprocessing steps to eliminate irrelevant, incomplete, or corrupted information, thus establishing a robust foundation for subsequent AI use. The data will serve as the backbone for training an AI model and chatbot, which will act as an interactive tool enabling users to ask complex, context-specific questions and receive accurate, validated answers derived from constrained literature. In parallel, the project incorporates a data structuring process supported by AI to organize sensor information from multiple sources into a standardized format. This structured data will include detailed sensor specifications, such as measurement range, applications, accuracy, and operating conditions, generated and documented with AI. These specifications will be systematically integrated into the sensor portal. To maintain the highest levels of accuracy and relevance, all AI-generated outputs will be reviewed and validated by subject matter experts (SMEs), with additional fields or parameters added as needed. Future stages of the project aim to expand the dataset beyond OSTI to include other sources and potentially incorporate unclassified controlled information (UCI) with restricted access protocols to address security and confidentiality requirements.

Mapes, NormanJ. [Idaho National Laboratory (INL), ↗
DOE OSTI2018

Spy the Lie: Detecting Malicious Insiders

Insider threat is a hard problem. There is no ground truth, there are innumerable variables, and the data is sparse. The types of crimes and abuses associated with insider threats are significant; the most serious include espionage, sabotage, terrorism, embezzlement, extortion, bribery, and corruption. Malicious activities include an even broader range of exploits, such as negligent use of classified data, fraud, cybercrime, unauthorized access to sensitive information, and illicit communications with unauthorized recipients. Inadvertent action or inaction without malicious intent (e.g., disposing of sensitive documents incorrectly) can also cause harm to an organization. This review article will explore insider threat, specifically behaviors, beliefs, and current debates within the field. Additionally particular focus is given to deception, a significant behavioral component of the malicious insider. Finally, research and policy implications for law enforcement and the intelligence community are addressed.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2020

NEON AOP Survey of Upper East River CO Watersheds: LAZ Files, LiDAR Surface Elevation, Terrain Elevation, and Canopy Height Rasters

Lawrence Berkeley National Laboratory (LBNL) contracted the National Ecological Observatory Network Airborne Observation Platform (NEON AOP) to observe watersheds of interest surrounding Crested Butte, CO with remotely sensed data, including LiDAR. The flight box design encompassed the watersheds, surveying a total area of 334 km2 across 72 lines. The instrument used was an Optech Gemini, with a pulse density of 2-9 pulses m-2 across the study area (see final report document for detailed information (Goulden and Musinsky 2020). These LiDAR data are the primary data that were provided by NEON, and include both LiDAR source data and derived surface products. The source data are contained in 'crbu_laz.zip' and 'classified_point_cloud.zip. 'classified_point_cloud.zip' represents an additional processing level on 'crbu_laz.zip'. The derived products are 'dsm_mosaic_min_phase_me.tif', 'dtm_mosaic_min_phase_me.tif', and 'tch_mosaic_min_phase_me.tif'. These files are raster mosaics of a digital surface elevation model (DSM), digital terrain elevation model (DTM), and a canopy height model (CHM) at 1 m resolution with a height threshold of > 2 m, respectively, in geotiff (.tif) format. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231. [UPDATE] On 06/26/2026 the files 'crbu_laz.zip' and 'classified_point_cloud.zip' were replaced with original output files from NEON to address zip file corruption and naming errors. No changes were made to the underlying data. (See Methods Step 2 for details).

2018 NEON and 2025 CHESS Campaigns↗
DOE OSTI2023

Leipzig University W-Band Cloud Radar, Gothic (Colorado), SAIL Campaign Second Winter (15.11.2022 - 05.06.2023)

The instrument is a polarimetric W-Band simultaneous transmission simultaneous reception (STSR) Doppler Cloud Radar manufactured by Radiometer Physics GmbH (RPG, instrument type RPG-FMCW-94-DP). It was deployed about 500m away from the ARM M1 facility in Gothic (CO) near the Ore House between Nov 15, 2022 and Jun 5, 2023 at an altitude of 2913 m. The Doppler cloud radar was mounted on a prototype of a cold temperature scanner manufactured by RPG. The dataset contains common radar variables like equivalent radar reflectivity factor, mean Doppler velocity, Doppler spectrum width as well as common polarimetric radar variables like differential phase shift (PhiDP), specific differential phase shift (KDP), correlation coefficient (RHV) and differential reflectivity (ZDR). Between Nov 15, 2022, and Feb 7, 2023 the radar was operated at 40° constant elevation and 151° azimuth (towards the KAZR at the AMF1). Between Feb 7 and Mar 15, 2023 the following scan cycle was repeated in 15min intervals: zenith pointing measurements, RHI scans from 0° to 90° and 90° to 0° elevation at 151° azimuth, constant elevation measurements at 40° elevation and 151° azimuth and PPI scans at 85° elevation for wind retrievals (one PPI per hour). Between Mar 15 18:18 UTC and Apr 8, 2023 11:04 UTC, the scanner was stuck at 72° elevation, which resulted in corrupted files for this period. For that period, the files containing data are uploaded but should be treated with care. Between Apr 8 and Apr 14, 2023 the scanning pattern described above was performed again. Afterwards, except for May 17 and May 18, 2023 where scanning patterns as described above were performed, measurements continued at 40° elevation and 151° azimuth until May 22, 2023 09:00 UTC. The scanning pattern was then started again, but without hourly PPI measurements until May 31, 2023. Between May 31 and Jun 5, 2023 solely zenith-pointing observations were performed. The data provided here are the RPG Level 1 (L1) files, Doppler spectra Level 0 (L0) files are available upon request.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2013

STUDIES IN ASTRONOMICAL TIME SERIES ANALYSIS. VI. BAYESIAN BLOCK REPRESENTATIONS

This paper addresses the problem of detecting and characterizing local variability in time series and other forms of sequential data. The goal is to identify and characterize statistically significant variations, at the same time suppressing the inevitable corrupting observational errors. We present a simple nonparametric modeling technique and an algorithm implementing it-an improved and generalized version of Bayesian Blocks-that finds the optimal segmentation of the data in the observation interval. The structure of the algorithm allows it to be used in either a real-time trigger mode, or a retrospective mode. Maximum likelihood or marginal posterior functions to measure model fitness are presented for events, binned counts, and measurements at arbitrary times with known error distributions. Problems addressed include those connected with data gaps, variable exposure, extension to piecewise linear and piecewise exponential representations, multivariate time series data, analysis of variance, data on the circle, other data modes, and dispersed data. Simulations provide evidence that the detection efficiency for weak signals is close to a theoretical asymptotic limit derived by Arias-Castro et al. In the spirit of Reproducible Research all of the code and data necessary to reproduce all of the figures in this paper are included as supplementary material.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2013

IDENTIFYING IONIZED REGIONS IN NOISY REDSHIFTED 21 cm DATA SETS

One of the most promising approaches for studying reionization is to use the redshifted 21 cm line. Early generations of redshifted 21 cm surveys will not, however, have the sensitivity to make detailed maps of the reionization process, and will instead focus on statistical measurements. Here, we show that it may nonetheless be possible to directly identify ionized regions in upcoming data sets by applying suitable filters to the noisy data. The locations of prominent minima in the filtered data correspond well with the positions of ionized regions. In particular, we corrupt semi-numeric simulations of the redshifted 21 cm signal during reionization with thermal noise at the level expected for a 500 antenna tile version of the Murchison Widefield Array (MWA), and mimic the degrading effects of foreground cleaning. Using a matched filter technique, we find that the MWA should be able to directly identify ionized regions despite the large thermal noise. In a plausible fiducial model in which {approx}20% of the volume of the universe is neutral at z {approx} 7, we find that a 500-tile MWA may directly identify as many as {approx}150 ionized regions in a 6 MHz portion of its survey volume and roughly determine the size of each of these regions. This may, in turn, allow interesting multi-wavelength follow-up observations, comparing galaxy properties inside and outside of ionized regions. We discuss how the optimal configuration of radio antenna tiles for detecting ionized regions with a matched filter technique differs from the optimal design for measuring power spectra. These considerations have potentially important implications for the design of future redshifted 21 cm surveys.

79 ASTRONOMY AND ASTROPHYSICS↗