SEARCH · Engineering Papers
Results for “ensemble Kalman filters”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Autodifferentiable Ensemble Kalman Filters
Data assimilation is concerned with sequentially estimating a temporally evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Here, our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble Kalman filters to scale to high-dimensional states and the power of automatic differentiation to train high-dimensional surrogate models for the dynamics. Numerical results using the Lorenz-96 model show that AD-EnKFs outperform existing methods that use expectation-maximization or particle filters to merge data assimilation and machine learning. In addition, AD-EnKFs are easy to implement and require minimal tuning.
A Stochastic Covariance Shrinkage Approach in Ensemble Transform Kalman Filtering
The Ensemble Kalman Filters (EnKF) employ a Monte-Carlo approach to represent covariance information, and are affected by sampling errors in operational settings where the number of model realizations is much smaller than the model state dimension. To alleviate the effects of these errors EnKF relies on model-specific heuristics such as covariance localization, which takes advantage of the spatial locality of correlations among the model variables. This work proposes an approach to alleviate sampling errors that utilizes a locally averaged-in-time dynamics of the model, described in terms of a climatological covariance of the dynamical system. We use this covariance as the target matrix in covariance shrinkage methods, and develop a stochastic covariance shrinkage approach where synthetic ensemble members are drawn to enrich both the ensemble subspace and the ensemble transformation. We additionally provide for a way in which this methodology can be localized similar to the state-of-the-art LETKF method, and that for a certain model setup, our methodology significantly outperforms it.
Multifidelity Ensemble Kalman Filtering Using Surrogate Models Defined by Theory-Guided Autoencoders
Data assimilation is a Bayesian inference process that obtains an enhanced understanding of a physical system of interest by fusing information from an inexact physics-based model, and from noisy sparse observations of reality. The multifidelity ensemble Kalman filter (MFEnKF) recently developed by the authors combines a full-order physical model and a hierarchy of reduced order surrogate models in order to increase the computational efficiency of data assimilation. The standard MFEnKF uses linear couplings between models, and is statistically optimal in case of Gaussian probability densities. This work extends the MFEnKF into to make use of a broader class of surrogate model such as those based on machine learning methods such as autoencoders non-linear couplings in between the model hierarchies. We identify the right-invertibility property for autoencoders as being a key predictor of success in the forecasting power of autoencoder-based reduced order models. We propose a methodology that allows us to construct reduced order surrogate models that are more accurate than the ones obtained via conventional linear methods. Numerical experiments with the canonical Lorenz'96 model illustrate that nonlinear surrogates perform better than linear projection-based ones in the context of multifidelity ensemble Kalman filtering. We additionality show a large-scale proof-of-concept result with the quasi-geostrophic equations, showing the competitiveness of the method with a traditional reduced order model-based MFEnKF.
Reduced-order autodifferentiable ensemble Kalman filters
This paper introduces a computational framework to reconstruct and forecast a partially observed state that evolves according to an unknown or expensive-to-simulate dynamical system. Our reduced-order autodifferentiable ensemble Kalman filters (ROAD-EnKFs) learn a latent low-dimensional surrogate model for the dynamics and a decoder that maps from the latent space to the state space. The learned dynamics and decoder are then used within an EnKF to reconstruct and forecast the state. Numerical experiments show that if the state dynamics exhibit a hidden low-dimensional structure, ROAD-EnKFs achieve higher accuracy at lower computational cost compared to existing methods. If such structure is not expressed in the latent state dynamics, ROAD-EnKFs achieve similar accuracy at lower cost, making them a promising approach for surrogate state reconstruction and forecasting.
Covariance operator estimation: Sparsity, lengthscale, and ensemble Kalman filters
This paper investigates covariance operator estimation via thresholding. For Gaussian random fields with approximately sparse covariance operators, we establish non-asymptotic bounds on the estimation error in terms of the sparsity level of the covariance and the expected supremum of the field. We prove that thresholded estimators enjoy an exponential improvement in sample complexity compared with the standard sample covariance estimator if the field has a small correlation lengthscale. As an application of the theory, we study thresholded estimation of covariance operators within ensemble Kalman filters.
Ensemble Kalman filter for data assimilation coupled with low-resolution computations techniques applied in fluid dynamics
This paper presents an innovative Reduced-order model (ROM) for merging experimental and simulation data using data assimilation (DA) to estimate the "True" state of a fluid dynamics system, leading to more accurate predictions. Our methodology introduces a novel approach by implementing the ensemble Kalman filter (EnKF) within a reduced-dimensional framework, grounded in a robust theoretical foundation and applied to fluid dynamics. To address the substantial computational demands of DA, the proposed ROM employs low-resolution (LR) techniques to drastically reduce computational costs. This innovative approach involves downsampling datasets for DA computations, followed by an advanced reconstruction technique based on low-cost singular value decomposition (lcSVD). The lcSVD method, a key innovation in this paper, has never been applied to DA before and offers a highly efficient way to enhance resolution with minimal computational resources. Our results demonstrate significant reductions in both computation time and RAM usage through these LR techniques without compromising the accuracy of the estimations. For instance, in a turbulent test case, for a data compression rate of 15.9, the LR approach can achieve a speed-up of 13.7 and a RAM compression of 90.9% while maintaining a low relative root mean square error (RRMSE) of 2.6%, compared to 0.8% in the high-resolution (HR) reference. Furthermore, we highlight the effectiveness of the EnKF in estimating and predicting the state of fluid flow systems based on limited observations and given low-fidelity numerical data. This paper highlights the potential of the proposed DA method in fluid dynamics applications, particularly for improving computational efficiency in CFD and related fields. Its ability to balance accuracy with low computational and memory costs makes it especially suitable for large-scale and real-time applications, such as environmental monitoring or engineering design. This method will be incorporated into ModelFLOWs-app.
TensorGraphicalModels: A Julia toolbox for multiway covariance models and ensemble Kalman filter
Not Available
Estimation of Power System Frequency Dynamics Using Cubature and Ensemble Kalman Filters
Explore the source record for details and available documents.
An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems
We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems with superior accuracy. A major drawback of existing filtering methods, e.g., particle filters or ensemble Kalman filters, is the low accuracy in handling high-dimensional and highly nonlinear problems. EnSF attacks this challenge by exploiting the score-based diffusion model, defined in a pseudo-temporal domain, to characterizing the evolution of the filtering density. EnSF stores the information of the recursively updated filtering density function in the score function, in stead of storing the information in a set of finite Monte Carlo samples (used in particle filters and ensemble Kalman filters). Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation that uses mini-batch-based Monte Carlo estimator to directly approximate the score function at any pseudo-spatial-temporal location, which provides sufficient accuracy in solving high-dimensional nonlinear problems as well as saves tremendous amount of time spent on training neural networks. Another essential aspect of EnSF is its analytical update step, gradually incorporating data information into the score function, which is crucial in mitigating the degeneracy issue faced when dealing with very high-dimensional nonlinear filtering problems. High-dimensional Lorenz systems are used to demonstrate the performance of our method. EnSF provides surprisingly impressive performance in reliably tracking extremely high-dimensional Lorenz systems (up to 1,000,000 dimension) with highly nonlinear observation processes, which is a well-known challenging problem for existing filtering methods.
Rock Physics-Based Data Assimilation of Integrated Continuous Active-Source Seismic and Pressure Monitoring Data during Geological Carbon Storage
Summary There has been substantial controversy concerning the role of geological carbon storage (GCS) in sequestering anthropogenic carbon emissions to mitigate climate change and global warming. Arguments center on the inability to monitor a geological storage site precisely and continuously, especially highlighting the associated costs and spatiotemporal trade-offs when using conventional subsurface monitoring techniques (well logs, core samples, chemical tracers, and 4D seismics). Active surveillance of GCS sites is essential for managing and mitigating potential leaks but is also required by regulation. With the goal of enhancing the monitoring capability at GCS sites, we present a rock physics-based joint data assimilation model to study a popular GCS site at Cranfield, Mississippi, USA. Synthetic continuous active-source seismic monitoring (CASSM) data (in the form of Vp and Qp measurements) and wellbore pressure monitoring data are assimilated with an ensemble of reservoir realizations to monitor gas saturation and reservoir pressure changes over a period of 100 years. Synthetic seismic attributes are generated using rock physics models (RPMs) and wellbore pressure monitoring data are extracted from the ground truth. Two assimilation methods, ensemble Kalman filter (EnKF) and ensemble Kalman smoother (EnKS), are tested in an observation system simulation experiment (OSSE) environment to assess the prediction accuracy of the individual and composite observation systems. The joint monitoring system achieves more accurate estimates of gas saturation and pressure, across the time span from start of injection to end of forecast, as compared to a single type of monitoring tool and irrespective of data assimilation algorithm choice. These results indicate that jointly assimilated data from two types of sensors (in this case, crosswell seismic and downhole pressure) may lead to a more risk-reducing monitoring design. One would expect that more data, vis-à-vis inclusion of a new sensor type, will improve the accuracy of any GCS monitoring system. However, from a practical standpoint, one important question is whether such a gain in accuracy is worth the additional cost associated with the new sensor. This paper focuses on quantifying the gain in accuracy, such that a practitioner can answer this question.
The Potential Benefits of Handling Mixture Statistics via a Bi-Gaussian EnKF: Tests With All-Sky Satellite Infrared Radiances
The meteorological characteristics of cloudy atmospheric columns can be very different from their clear counterparts. Thus, when a forecast ensemble is uncertain about the presence/absence of clouds at a specific atmospheric column (i.e., some members are clear while others are cloudy), that column's ensemble statistics will contain a mixture of clear and cloudy statistics. Such mixtures are inconsistent with the ensemble data assimilation algorithms currently used in numerical weather prediction. Hence, ensemble data assimilation algorithms that can handle such mixtures can potentially outperform currently used algorithms. In this study, we demonstrate the potential benefits of addressing such mixtures through a bi-Gaussian extension of the ensemble Kalman filter (BGEnKF). The BGEnKF is compared against the commonly used ensemble Kalman filter (EnKF) using perfect model observing system simulated experiments (OSSEs) with a realistic weather model (the Weather Research and Forecast model). Synthetic all-sky infrared radiance observations are assimilated in this study. In these OSSEs, the BGEnKF outperforms the EnKF in terms of the horizontal wind components, temperature, specific humidity, and simulated upper tropospheric water vapor channel infrared brightness temperatures. This study is one of the first to demonstrate the potential of a Gaussian mixture model EnKF with a realistic weather model. Our results thus motivate future research toward improving numerical Earth system predictions though explicitly handling mixture statistics.
Physics-Informed Gaussian Process Regression for States Estimation and Forecasting in Power Grids
Real-time state estimation and forecasting are critical for the efficient operation of power grids. In this paper, a physics-informed Gaussian process regression (PhI-GPR) method is presented and used for forecasting and estimating the phase angle, angular speed, and wind mechanical power of a three-generator power grid system using sparse measurements. In standard data-driven Gaussian process regression (GPR), parameterized models for the prior statistics are fit by maximizing the marginal likelihood of observed data. In the PhI-GPR method, we propose to compute the prior statistics offline by solving stochastic differential equations (SDEs) governing the power grid dynamics. The short-term forecast of a power grid system dominated by wind generation is complicated by the stochastic nature of the wind and the resulting uncertainty in wind mechanical power. Here, we assume that the power grid dynamics are governed by swing equations, with the wind mechanical power fluctuating randomly in time. We solve these equations for the mean and covariances of the power grid states using the Monte Carlo simulation method. We demonstrate that the proposed PhI-GPR method can accurately forecast and estimate observed and unobserved states. For the considered problem, PhI-GPR has computational advantages over the ensemble Kalman filter (EnKF) method: In PhI-GPR, ensembles are computed offline and independently of the data acquisition process, whereas for EnFK, ensembles are computed online with data acquisition, rendering real-time forecast more challenging. We also demonstrate that the PhI-GPR forecast is more accurate than the EnKF forecast when the random mechanical wind power is non-Markovian. In contrast, the two methods produce similar forecasts for the Markovian mechanical wind power. For observed states, we show that PhI-GPR provides a forecast comparable to the standard data-driven GPR; both forecasts are significantly more accurate than the autoregressive integrated moving average (ARIMA) forecast. We also show that the ARIMA forecast is more sensitive to observation frequency and measurement errors than the PhI-GPR forecast.
Assimilation of X-band Phased-Array Radar Data with EnKF for the Analysis and Warning Forecast of a Tornadic storm
The impact of assimilating China's operational X-band Phased-Array radar's (X-PAR) data on the analysis and warning forecast of the vortex structure and intensity of the June 8, 2018 Foshan, Guangdong province, tornadic storm was investigated for the first time using an Ensemble Kalman Filter (EnKF) data assimilation system. Both radar radial velocity (Vr) and reflectivity (Z) from two S-band operational radars and one X-PAR were assimilated. Deterministic forecasts were launched every 6 min from 05:42 UTC (20 min before the tornado touched down) to 06:00 UTC from the EnKF mean analysis field. Five experiments were conducted to examine the added capability of Z assimilation of the EnKF system, and to investigate the impact of assimilating X-PAR data on the analysis and prediction of the tornadic storm. Compared to the experiment without Z assimilation, the assimilation of Z reduced the analysis error and greatly reduced the forecast error of Z. Furthermore, the assimilation of X-PAR data greatly improved the vortex structure of the tornadic storm at low levels, and improved the intensity of the rear inflow of the tornadic storm, especially with a higher assimilation frequency. Compared to the experiments without X-PAR data assimilation, assimilating X-PAR data improved the predictability of tornadic storm.
Joint state-parameter estimation for the reduced fracture model via the united filter
Here, in this paper, we introduce an effective United Filter method for jointly estimating the solution state and physical parameters in flow and transport problems within fractured porous media. Fluid flow and transport in fractured porous media are critical in subsurface hydrology, geophysics, and reservoir geomechanics. Reduced fracture models, which represent fractures as lower-dimensional interfaces, enable efficient multi-scale simulations. However, reduced fracture models also face accuracy challenges due to modeling errors and uncertainties in physical parameters such as permeability and fracture geometry. To address these challenges, we propose a United Filter method, which integrates the Ensemble Score Filter (EnSF) for state estimation with the Direct Filter for parameter estimation. EnSF, based on a score-based diffusion model framework, produces ensemble representations of the state distribution without deep learning. Meanwhile, the Direct Filter, a recursive Bayesian inference method, estimates parameters directly from state observations. The United Filter combines these methods iteratively: EnSF estimates are used to refine parameter values, which are then fed back to improve state estimation. Numerical experiments demonstrate that the United Filter method surpasses the state-of-the-art Augmented Ensemble Kalman Filter, delivering more accurate state and parameter estimation for reduced fracture models. This framework also provides a robust and efficient solution for PDE-constrained inverse problems with uncertainties and sparse observations.
Development of an open-source regional data assimilation system in PEcAn v. 1.7.2: application to carbon cycle reanalysis across the contiguous US using SIPNET
Abstract. The ability to monitor, understand, and predict the dynamics of the terrestrial carbon cycle requires the capacity to robustly and coherently synthesize multiple streams of information that each provide partial information about different pools and fluxes. In this study, we introduce a new terrestrial carbon cycle data assimilation system, built on the PEcAn model–data eco-informatics system, and its application for the development of a proof-of-concept carbon “reanalysis” product that harmonizes carbon pools (leaf, wood, soil) and fluxes (GPP, Ra, Rh, NEE) across the contiguous United States from 1986–2019. We first calibrated this system against plant trait and flux tower net ecosystem exchange (NEE) using a novel emulated hierarchical Bayesian approach. Next, we extended the Tobit–Wishart ensemble filter (TWEnF) state data assimilation (SDA) framework, a generalization of the common ensemble Kalman filter which accounts for censored data and provides a fully Bayesian estimate of model process error, to a regional-scale system with a calibrated localization. Combined with additional workflows for propagating parameter, initial condition, and driver uncertainty, this represents the most complete and robust uncertainty accounting available for terrestrial carbon models. Our initial reanalysis was run on an irregular grid of ∼ 500 points selected using a stratified sampling method to efficiently capture environmental heterogeneity. Remotely sensed observations of aboveground biomass (Landsat LandTrendr) and leaf area index (LAI) (MODIS MOD15) were sequentially assimilated into the SIPNET model. Reanalysis soil carbon, which was indirectly constrained based on modeled covariances, showed general agreement with SoilGrids, an independent soil carbon data product. Reanalysis NEE, which was constrained based on posterior ensemble weights, also showed good agreement with eddy flux tower NEE and reduced root mean square error (RMSE) compared to the calibrated forecast. Ultimately, PEcAn's new open-source regional data assimilation framework provides a scalable workflow for harmonizing multiple data constraints and providing a uniform synthetic platform for carbon monitoring, reporting, and verification (MRV) as well as accelerating terrestrial carbon cycle research.