Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Missing data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

Imputation of urban environmental sensor data using gated attention bidirectional long short-term memory (GA-BiLSTM): methods, performance, and implications

Urban environmental monitoring networks frequently encounter significant data gaps due to sensor malfunctions, environmental disturbances, and communication failures. Reliable approaches to address these gaps are essential for ensuring the continuity and quality of environmental data streams. In this study, we developed a gated attention bidirectional long short-term memory (GA-BiLSTM) model to impute missing data in a dense urban monitoring network. Using observations from the CROCUS network in Chicago, we evaluated GA-BiLSTM against widely used approaches (XGBoost and K-nearest neighbors) under scenarios of both short-term intermittent gaps and prolonged outages. GA-BiLSTM consistently outperformed comparative methods, particularly during extended outages of up to ten days, demonstrating its ability to capture spatiotemporal dependencies across sensor nodes. Beyond performance metrics, feature importance and spatial network analyses highlighted the unexpected but critical predictive role of peripheral rural nodes, underlining their strategic value for maintaining robust urban monitoring systems. These results emphasize that advanced imputation methods can substantially improve the reliability of environmental monitoring networks and support more resilient data infrastructures for urban sustainability.

Data imputation↗

Soil Water Percolation Flux, April 2017 to March 2019, BR-Ma2, Manaus

Soil water percolation flux measurements derived from six passive wick flux meters across three topographic positions (valley, slope and plateau). Data has been processed to compute the mean for each topographic position where two flux meters were installed at each topographic position. Missing data represented a few percent of the total data set and were filled using two approaches. In the event one of the flux meters malfunctioned, we used data from the other sensor from the same topographic position without averaging. When both sensors were not working, such as during the period from September to October 2017 for the slope and valley and July 2018 for the plateau, data was filled using a nonlinear regression model that was based on local precipitation data and coefficients that were empirically derived for each topographic position. The metadata tab included in the .csv has additional information on locations where the sensors were installed, and other installation/maintenance details. Contact ksolander@lanl.gov if you need to use this dataset for additional information.

54 ENVIRONMENTAL SCIENCES↗

Frost Tables, NGEE Areas B, C and D, Utqiagvik (Barrow), Alaska, 2012-2014

This dataset represents spatially intensive thaw depth surveys with individual point measurements spaced ~0.5 m apart. The three ~10x10m grids cover an ice wedge and a portion of its two neighboring polygons. The data contain thaw depth, frost table elevation, ground surface elevation, active layer depth and surface water inundation across three seasons (2012, 2013 and 2014) at Utqiagvik (Barrow) NGEE Areas B, C and D. There are a total of 69 extracted files separated by month, year, and area. The data were originally provided in excel workbooks, these files have been preserved within the zip "Original_Files." The files were transformed for archiving with images pulled out into JPEGs, data preserved as CSV files and some methodological information preserved as TXT files. Some functions, features, and formatting were lost during the transformation. Missing data were defined with blank cells. Additional documentation can be found in files labelled with "information.txt" file endings.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).[2024-07-15] Renamed BEO_Thaw_Depth_Frost_Table_Area_*_*_*.csv, BEO_Thaw_Depth_Frost_Table_Area_*_*_*_Plot_1.JPEG, and BEO_Thaw_Depth_Frost_Table_Area_*_*_*_Plot_2.JPEG names that had E or L to be Early or Late. Renamed BEO_Thaw_Depth_Frost_Table_Area_*_*_*_Plot_1.JPEG, BEO_Thaw_Depth_Frost_Table_Area_*_*_*_Plot_2.JPEG, and BEO_Thaw_Depth_Frost_Table_Area_*_information_Plot_1.JPEG to BEO_Thaw_Depth_Frost_Table_Area_*_*_*_Gridpoints.JPEG, BEO_Thaw_Depth_Frost_Table_Area_*_*_*_3Dgraphic.JPEG, and BEO_Thaw_Depth_Frost_Table_Area_*_information_LiDAR_DEM.JPEG, respectively. Updated BEO_Thaw_Depth_Frost_Table_flmd.csv file to reflect above updates.

54 ENVIRONMENTAL SCIENCES↗

Ground Water Levels for NGEE Areas A, B, C and D, Utqiagvik (Barrow), Alaska, 2012-2014

Ice wedge polygonal tundra water levels were measured at a total of 45 locations representing polygon centers and troughs during three summers. Early season water levels, which were still affected by ice and snow, are represented by manual measurements only. Continuous (less than hourly) measurements followed through early fall (~mid-Sep). The data set contains inundation depth (cm), absolute water level and local ground surface elevation (masl). There are a total of 209 files in this dataset. The data were originally provided in Excel workbooks and these files have been preserved within the zip "Original_Files." The workbooks were also transformed and separated by well ID, year, and/or area into 205 extracted files. The Excel files were transformed for archiving with images pulled out into JPEGs, data preserved as CSV files and some methodological information preserved as TXT files. Some functions, features, and formatting were lost during the transformation. Missing data were defined with blank cells. Additional documentation can be found in files labelled with "information.txt" file endings.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

Benchmark probabilistic solar forecasts: Characteristics and recommendations

We illustrate and compare commonly used benchmark, or reference, methods for probabilistic solar forecasting that researchers use to measure the performance of their proposed techniques. A thorough review of the literature indicates wide variation in the benchmarks implemented in probabilistic solar forecast studies. To promote consistent and sensible methodological comparisons, we implement and compare ten variants from six common benchmark classes at two temporal scales: intra-hourly forecasts and hourly resolution forecasts. Using open-source Surface Radiation Budget Network (SURFRAD) data from 2018, these benchmark methods are compared using proper probabilistic metrics and common diagnostic tools. Practical implementation issues, such as the impact of missing data and applicability for operational forecasting, are also discussed. Furthermore, we make recommendations for practitioners on the appropriate selection of benchmark methods to properly showcase state-of-the-art improvements in forecast reliability and sharpness. All code and open-source data are available on Github for reproducibility and for other researchers to apply the same benchmark methods to their own data.

14 SOLAR ENERGY↗

Deep learning-based spatio-temporal estimate of greenhouse gas emissions using satellite data

Accurate estimation of greenhouse gases (GHGs) emissions is very important for developing mitigation strategies to climate change by controlling and reducing GHG emissions. This project aims to develop multiple deep learning approaches to estimate anthropogenic greenhouse gas emissions using multiple types of satellite data. NO2 concentration is chosen as an example of GHGs to evaluate the proposed approach. Two sentinel satellites (sentinel-2 and sentinel-5P) provide multiscale observations of GHGs from 10-60m resolution (sentinel-2) to ~kilometer scale resolution (sentinel-5P). Among multiple deep learning (DL) architectures evaluated, two best DL models demonstrate that key features of spatio-temporal satellite data and additional information (e.g., observation times and/or coordinates of ground stations) can be extracted using convolutional neural networks and feed forward neural networks, respectively. In particular, irregular time series data from different NO 2 observation stations limit the flexibility of long short-term memory architecture, requiring zero-padding to fill in missing data. However, deep neural operator (DNO) architecture can stack time-series data as input, providing the flexibility of input structure without zero-padding. As a result, the DNO outperformed other deep learning architectures to account for time-varying features. Overall, temporal patterns with smooth seasonal variations were predicted very well, while frequent fluctuation patterns were not predicted well. In addition, uncertainty quantification using conformal inference method is performed to account for prediction ranges. Overall, this research will lead to a new groundwork for estimating greenhouse gas concentrations using multiple satellite data to enhance our capability of tracking the cause of climate change and developing mitigation strategies.

54 ENVIRONMENTAL SCIENCES↗

The significant solar proton events in 20th solar cycle for the period October 1964 to March 1970

Solar proton data are presented from observations by the Explorer 21, 28, 34 and 41 satellites. The NASA Solar Particle Alert Network (SPAN) solar optical and radio frequency data for the period May 1967 to March 1970 are associated with the proton events observed by the Explorer 34 and 41 satellites; however, missing data are supplemented with data recorded at other international observatories. From a radiation hazard standpoint, NASA is concerned with solar proton events of the order of 10 to the 8th power proton/sq cm. Radiation dose data are presented for some of the large proton events that have occurred thus far in the 20th solar cycle and are compared with some of the large proton events of the 19th solar cycle. Finally, the results of a simple parametric correlation study are presented for both the 19th and 20th solar cycles.

Atwell, W.↗

Effective Interpolation of Incomplete Satellite-Derived Leaf-Area Index Time Series for the Continental United States

Many earth science modeling applications employ continuous input data fields derived from satellite data. Environmental factors, sensor limitations and algorithmic constraints lead to data products of inherently variable quality. This necessitates interpolation of one form or another in order to produce high quality input fields free of missing data. The present research tests several interpolation techniques as applied to satellite-derived leaf area index, an important quantity in many global climate and ecological models. The study evaluates and applies a variety of interpolation techniques for the Moderate Resolution Imaging Spectroradiometer (MODIS) Leaf-Area Index Product over the time period 2001-2006 for a region containing the conterminous United States. Results indicate that the accuracy of an individual interpolation technique depends upon the underlying land cover. Spatial interpolation provides better results in forested areas, while temporal interpolation performs more effectively over non-forest cover types. Combination of spatial and temporal approaches offers superior interpolative capabilities to any single method, and in fact, generation of continuous data fields requires a hybrid approach such as this.

Jasinski, Michael F.↗

Characterizing Sub-Cohorts via Data Normalization and Representation Learning

The process of identifying a cohort of interest is a very challenging task. It requires manually inspecting many patient records of complex structure that might include medical coding errors and missing data. This paper presents a computational pipeline for refining the process of cohort selection based on medical concepts recorded in the electronic health records (EHRs). The pipeline extracts EHR data for a given cohort and normalizes this data using standard vocabularies. Then a stacked denoising autoencoder is used to embed the normalized patient vectors in a low dimensional space, where the patients are subsequently clustered into sub-cohorts. The goal is to represent the cohort in a standard format and abstract variants of sub-populations. As a use-case, we applied the pipeline to 1.8 million Veterans diagnosed with major depressive disorder (MDD), and identified four meaningful sub-cohorts using the features learned by the autoencoder. Then, each sub-cohort was explored using a set of keywords for interpretation.

Rush III, Everett↗

Automated Dynamic Demand Response Implementation on a Micro-grid

In this paper, we describe a system for real-time automated Dynamic and Sustainable Demand Response with sparse data consumption prediction implemented on the University of Southern California campus microgrid. Supply side approaches to resolving energy supply-load imbalance do not work at high levels of renewable energy penetration. Dynamic Demand Response (D 2 R) is a widely used demand-side technique to dynamically adjust electricity consumption during peak load periods. Our D 2 R system consists of accurate machine learning based energy consumption forecasting models that work with sparse data coupled with fast and sustainable load curtailment optimization algorithms that provide the ability to dynamically adapt to changing supply-load imbalances in near real-time. Our Sustainable DR (SDR) algorithms attempt to distribute customer curtailment evenly across sub-intervals during a DR event and avoid expensive demand peaks during a few sub-intervals. It also ensures that each customer is penalized fairly in order to achieve the targeted curtailment. We develop near linear-time constant-factor approximation algorithms along with Polynomial Time Approximation Schemes (PTAS) for SDR curtailment that minimizes the curtailment error defined as the difference between the target and achieved curtailment values. Our SDR curtailment problem is formulated as an Integer Linear Program that optimally matches customers to curtailment strategies during a DR event while also explicitly accounting for customer strategy switching overhead as a constraint. We demonstrate the results of our D 2 R system using real data from experiments performed on the USC smartgrid and show that 1) our prediction algorithms can very accurately predict energy consumption even with noisy or missing data and 2) our curtailment algorithms deliver DR with extremely low curtailment errors in the 0.01-0.05 kWh range.

Kuppannagari, Sanmukh R.↗

COMPASS-FME Synoptic Sites Level 1 Sensor Data v1-2

This is the version 1-2 Level 1 (L1) data release for COMPASS-FME environmental sensors located at our synoptic field sites. COMPASS-FME is studying sites in two distinct regions, the Chesapeake Bay and the Western Lake Erie Basin. We established the network at seven "synoptic" (observational) sites along the Chesapeake Bay and Lake Erie coastlines, collectively generating over three million observations per month, to track and comprehend environmental changes where land and water intersect. Additionally, the two regions provide an interesting contrast of saltwater and freshwater coasts that allow us to differentiate the impacts of inundation and coastal water chemistries in two nationally important coastal systems.L1 data are close to raw, but are units-transformed and have out-of-instrument-bounds and out-of-service flags added. Duplicates and missing data are removed but otherwise these data are not filtered, and have not been subject to any additional algorithmic or human QA/QC. Any scientific analyses of L1 data should be performed with care. **This dataset will be updated quarterly with new data for the duration of the project**This dataset includes:- An overall dataset README file that describes the current version, gives citation and contact information, etc.- Site- and year-specific folders, each holding up to 12 CSV (comma separated value) data files for each site and plot in that year.- Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, as well as a general description of the site.- Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are normally logged every 15 minutes. Please see v1-2 Synoptic L1 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning.

54 ENVIRONMENTAL SCIENCES↗

COMPASS-FME Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) Experiment Level 1 Sensor Data v1-2

This is the version 1-2 Level 1 (L1) data release for COMPASS-FME environmental sensors located at our Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) experimental site. This manipulative, ecosystem-scale TEMPEST experiment addresses the potential for freshwater and estuarine-water disturbance events to alter tree function, species composition, and ecosystem processes in a deciduous coastal forest in MD, USA. The experiment uses a large-unit (2000 m2), un-replicated experimental design, with three 50 m × 40 m plots serving as control, freshwater, and estuarine-water treatments.L1 data are close to raw, but are units-transformed and have out-of-instrument-bounds and out-of-service flags added. Duplicates and missing data are removed but otherwise these data are not filtered, and have not been subject to any additional algorithmic or human QA/QC. Any scientific analyses of L1 data should be performed with care. **This dataset will be updated quarterly with new data for the duration of the project**This dataset includes:- An overall dataset README file that describes the current version, gives citation and contact information, etc.- Site- and year-specific folders, each holding up to 12 CSV (comma separated value) data files for each site and plot in that year.- Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, detailed flood times, as well as a general description of the site.- Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are normally logged every 15 minutes.Please see v1-2 TEMPEST L1 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning.The TEMPEST flood events occurred on the following dates. They lasted for ~10 hours each day and delivered ~80,000 gallons to each plot; many data streams are available at 1 or 5 minute frequency during these periods.* Tests: Aug 25 (fresh plot) and Sep 9 (salt plot), 2021* TEMPEST 1: June 22, 2022* TEMPEST 2: June 6-7, 2023* TEMPEST 3: June 11-13, 2024

54 ENVIRONMENTAL SCIENCES↗

Inferring Netflow Data

Netflow data is a specific format for looking at network traffic. With this format, many security tests are easy to run. Cloud Service Providers produce logs that do not contain all of the content that Netflow data contains. There is a possibility that the fields that are in Netflow data but absent in Cloud Service Provider logs can be inferred from the fields that are present in the Cloud Service Provider logs. By inferring the missing data fields, the same test that can be run with Netflow data would be possible to run on the logs produced by Cloud Service Providers. Cloud Service Providers include Amazon Web Services and Box.com. There are multiple different types of Cloud Services, and each provider handles them differently and produces different logs. There are IaaS (meaning infrastructure as a service), Saas (meaning software as a service), and Paas (meaning platform as a service). Each provides a different use for the user.

97 MATHEMATICS AND COMPUTING↗

Analysis of space radiation data of semiconductor memories

This article presents an analysis of radiation effects for several select device types and technologies aboard the Combined Release and Radiation Effects Satellite (CRRES) satellite. These space-flight measurements covered a period of about 14 months of mission lifetime. Single Event Upset (SEU) data of the investigated devices from the Microelectronics Package (MEP) were processed and analyzed. Valid upset measurements were determined by correcting for invalid readings, hard failures, missing data tapes (thus voids in data), and periods over which devices were disabled from interrogation. The basic resolution time of the measurement system was confirmed to be 2 s. Lessons learned, important findings, and recommendations are presented.

Flight Experiment↗

Phase Identification in Real Distribution Networks with High PV Penetration Using Advanced Metering Infrastructure Data

Many distribution network monitoring and control applications - including state estimation, volt/VAR optimization, and network reconfiguration - rely on accurate network models; however, the network models maintained by utilities can become outdated because of restoration activities, network reconfiguration, and missing data. With the widespread deployment of advanced metering infrastructure (AMI), abundant measurement data from low-voltage secondary networks are available. The AMI measurement data can be used for phase identification to improve the network models. Although the existing phase identification techniques work well in passive distribution feeders that do not have photovoltaic (PV) generation, they can fail to accurately identify the phases in the presence of PV. This paper proposes a robust phase identification algorithm based on supervised machine learning that accurately identifies the AMI meter phase connectivity in the presence of significant PV generation. The proposed algorithm does not require network topology information or feeder head measurement data. The algorithm is validated using the AMI measurement data collected in the field and the field-validated phase connectivity database on two real distribution feeders from San Diego Gas & Electric Company that have significant PV generation.

advanced metering infrastructure↗

Phase Identification in Real Distribution Networks with High PV Penetration Using Advanced Metering Infrastructure Data

Many distribution network monitoring and control applications - including state estimation, Volt/VAr optimization, and network reconfiguration - rely on accurate network models; however, the network models maintained by utilities can become outdated because of restoration activities, network reconfiguration, and missing data. With the widespread deployment of advanced metering infrastructure (AMI), abundant measurement data from low-voltage secondary networks are available. The AMI measurement data can be used for phase identification to improve the network models. Although the existing phase identification techniques work well in passive distribution feeders that do not have photovoltaic (PV) generation, they can fail to accurately identify the phases in the presence of PV. This paper proposes a robust phase identification algorithm based on supervised machine learning that accurately identifies the AMI meter phase connectivity in the presence of significant PV generation. The proposed algorithm does not require network topology information or feeder-head measurement data. The algorithm is validated using the AMI measurement data collected in the field and the field-validated phase connectivity database on two real distribution feeders from San Diego Gas & Electric Company that have significant PV generation.

advanced metering infrastructure (AMI)↗

ALPHANSO: Open-source modeling of (α, n) neutron source terms

Applications ranging from nuclear safeguards to dark matter detection require accurate predictions of neutron yields and energy spectra produced by (α, n) reactions. Legacy tools like SOURCES-4C remain widely used despite significant limitations, including outdated nuclear data, missing target nuclides, and restricted accessibility. Here, we present ALPHANSO, an open-source Python package for calculating (α, n) neutron source terms. ALPHANSO incorporates modern nuclear data libraries and formats covering all naturally occurring target nuclides and provides a transparent, modular framework for updating or extending the data as new evaluations are released. Comparison with an updated version of SOURCES-4A, NeuCBOT, and experimental measurements across a range of elements and materials shows that ALPHANSO reproduces neutron yields and spectra in good agreement with experimental data and state-of-the-art (α, n) calculations. These results demonstrate that ALPHANSO is a reliable, accessible, and modern alternative to legacy (α, n) source term codes such as SOURCES-4C. Its open-source design and modular data handling make it readily extensible to future evaluated nuclear data and low-background applications.

(α, n) reactions↗

COMPASS-FME Synoptic Sites Level 1 Sensor Data v2-1

This is the version 2-1 Level 1 (L1) data release for COMPASS-FME environmental sensors located at our synoptic field sites. COMPASS-FME is studying sites in two distinct regions, the Chesapeake Bay and the Western Lake Erie Basin. We established the network at seven "synoptic" (observational) sites along the Chesapeake Bay and Lake Erie coastlines, collectively generating over three million observations per month, to track and comprehend environmental changes where land and water intersect. Additionally, the two regions provide an interesting contrast of saltwater and freshwater coasts that allow us to differentiate the impacts of inundation and coastal water chemistries in two nationally important coastal systems. L1 data are close to raw, but are units-transformed and have out-of-instrument-bounds, out-of-service, and outlier flags added. Duplicates and missing data are removed but otherwise these data are not filtered, and have not been subject to any additional algorithmic or human QA/QC. Any scientific analyses of L1 data should be performed with care. **This dataset will be updated quarterly with new data for the duration of the project** This dataset includes: - An overall dataset README file that describes the current version, gives citation and contact information, etc. - Site- and year-specific folders, each holding up to 12 CSV (comma separated value) data files for each site and plot in that year. - Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, as well as a general description of the site. - Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are normally logged every 15 minutes. Please see v2-0 Synoptic L1 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning. This dataset was updated 2026-03-12: (i) data now go through 2025-12-31 (previous end was 2025-06-30) and (ii) dataset and file names updated to “…v2-1” (previously was “v2-0”).

54 ENVIRONMENTAL SCIENCES↗