Engineering Papers⌕ Search

Engineering topics

Biggerstaff, Matthew

Publications and source records attributed to Biggerstaff, Matthew.

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

Background: The importance of infectious disease epidemic forecasting and prediction research is underscored by decades of communicable disease outbreaks, including COVID-19. Unlike other fields of medical research, such as clinical trials and systematic reviews, no reporting guidelines exist for reporting epidemic forecasting and prediction research despite their utility. We therefore developed the EPIFORGE checklist, a guideline for standardized reporting of epidemic forecasting research. Methods and findings: We developed this checklist using a best-practice process for development of reporting guidelines, involving a Delphi process and broad consultation with an international panel of infectious disease modelers and model end users. The objectives of these guidelines are to improve the consistency, reproducibility, comparability, and quality of epidemic forecasting reporting. The guidelines are not designed to advise scientists on how to perform epidemic forecasting and prediction research, but rather to serve as a standard for reporting critical methodological details of such studies. Conclusions: These guidelines have been submitted to the EQUATOR network, in addition to hosting by other dedicated webpages to facilitate feedback and journal endorsement.

60 APPLIED LIFE SCIENCES↗

Forecasting influenza activity using machine-learned mobility map

Human mobility is a primary driver of infectious disease spread. However, existing data is limited in availability, coverage, granularity, and timeliness. Data-driven forecasts of disease dynamics are crucial for decision-making by health officials and private citizens alike. In this work, we focus on a machine-learned anonymized mobility map (hereon referred to as AMM) aggregated over hundreds of millions of smartphones and evaluate its utility in forecasting epidemics. We factor AMM into a metapopulation model to retrospectively forecast influenza in the USA and Australia. We show that the AMM model performs on-par with those based on commuter surveys, which are sparsely available and expensive. We also compare it with gravity and radiation based models of mobility, and find that the radiation model’s performance is quite similar to AMM and commuter flows. Additionally, we demonstrate our model’s ability to predict disease spread even across state boundaries. Our work contributes towards developing timely infectious disease forecasting at a global scale using human mobility datasets expanding their applications in the area of infectious disease epidemiology.

60 APPLIED LIFE SCIENCES↗

Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.

Seasonal influenza results in substantial annual morbidity and mortality in the United States and worldwide. Accurate forecasts of key features of influenza epidemics, such as the timing and severity of the peak incidence in a given season, can inform public health response to outbreaks. As part of ongoing efforts to incorporate data and advanced analytical methods into public health decision-making, the United States Centers for Disease Control and Prevention (CDC) has organized seasonal influenza forecasting challenges since the 2013/2014 season. In the 2017/2018 season, 22 teams participated. A subset of four teams created a research consortium called the FluSight Network in early 2017. During the 2017/2018 season they worked together to produce a collaborative multi-model ensemble that combined 21 separate component models into a single model using a machine learning technique called stacking. This approach creates a weighted average of predictive densities where the weight for each component is determined by maximizing overall ensemble accuracy over past seasons. In the 2017/2018 influenza season, one of the largest seasonal outbreaks in the last 15 years, this multi-model ensemble performed better on average than all individual component models and placed second overall in the CDC challenge. It also outperformed the baseline multi-model ensemble created by the CDC that took a simple average of all models submitted to the forecasting challenge. This project shows that collaborative efforts between research teams to develop ensemble forecasting approaches can bring measurable improvements in forecast accuracy and important reductions in the variability of performance from year to year. Efforts such as this, that emphasize real-time testing and evaluation of forecasting models and facilitate the close collaboration between public health officials and modeling researchers, are essential to improving our understanding of how best to use forecasts to improve public health response to seasonal and emerging epidemic threats.

59 BASIC BIOLOGICAL SCIENCES↗

A systematic review and evaluation of Zika virus forecasting and prediction research during a public health emergency of international concern

Researchers published many studies which sought to predict and forecast important features of Zika virus (ZIKV) infections and their spread during the 2016–2017 ZIKV pandemic. We conducted a comprehensive review of such ZIKV prediction studies and evaluated their aims, the data sources they used, which methods were used, how timely they were published, and whether they provided sufficient information to be used or reproduced by others. Of the 73 studies evaluated, we found the accessibility, reproducibility, timeliness, and incorporation of uncertainty in these published predictions varied; indicating there is substantial room for improvement. We identified that the release of study findings before formal journal publication (‘pre-prints’) increased the timeliness of Zika prediction studies, but note they were infrequently used during this public health emergency. Addressing these areas can improve our understanding of Zika and other outbreaks and ensure forecasts can inform preparedness and response to future outbreaks, epidemics, and pandemics.

59 BASIC BIOLOGICAL SCIENCES↗

Reply to Bracher: Scoring probabilistic forecasts to maximize public health interpretability

Evaluating probabilistic forecasts in the context of a real-time public health surveillance system is a complicated business. We agree with Bracher’s (1) observations that the scores established by the US Centers for Disease Control and Prevention (CDC) and used to evaluate our forecasts of seasonal influenza in the United States are not “proper” by definition (2). We thank him for raising this important issue.

60 APPLIED LIFE SCIENCES↗

School dismissal as a pandemic influenza response: When, where and for how long?

We used individual-based computer simulation models at community, regional and national levels to evaluate the likely impact of coordinated pre-emptive school dismissal policies during an influenza pandemic. Such policies involve three key decisions: when, over what geographical scale, and how long to keep schools closed. Our evaluation includes uncertainty and sensitivity analyses, as well as model output uncertainties arising from variability in serial intervals and presumed modifications of social contacts during school dismissal periods. During the period before vaccines become widely available, school dismissals are particularly effective in delaying the epidemic peak, typically by 4–6 days for each additional week of dismissal. Assuming the surveillance is able to correctly and promptly diagnose at least 5–10% of symptomatic individuals within the jurisdiction, dismissals at the city or county level yield the greatest reduction in disease incidence for a given dismissal duration for all but the most severe pandemic scenarios considered here. Broader (multi-county) dismissals should be considered for the most severe and fast-spreading (1918-like) pandemics, in which multi-month closures may be necessary to delay the epidemic peak sufficiently to allow for vaccines to be implemented.

59 BASIC BIOLOGICAL SCIENCES↗

A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States

Influenza infects an estimated 9–35 million individuals each year in the United States and is a contributing cause for between 12,000 and 56,000 deaths annually. Seasonal outbreaks of influenza are common in temperate regions of the world, with highest incidence typically occurring in colder and drier months of the year. Real-time forecasts of influenza transmission can inform public health response to outbreaks. We present the results of a multiinstitution collaborative effort to standardize the collection and evaluation of forecasting models for influenza in the United States for the 2010/2011 through 2016/2017 influenza seasons. For these seven seasons, we assembled weekly real-time forecasts of seven targets of public health interest from 22 different models. We compared forecast accuracy of each model relative to a historical baseline seasonal average. Across all regions of the United States, over half of the models showed consistently better performance than the historical baseline when forecasting incidence of influenza-like illness 1 wk, 2 wk, and 3 wk ahead of available data and when forecasting the timing and magnitude of the seasonal peak. In some regions, delays in data reporting were strongly and negatively associated with forecast accuracy. More timely reporting and an improved overall accessibility to novel and traditional data sources are needed to improve forecasting accuracy and its integration with real-time public health decision making.

60 APPLIED LIFE SCIENCES↗

Results from the centers for disease control and prevention’s predict the 2013–2014 Influenza Season Challenge

Background: Early insights into the timing of the start, peak, and intensity of the influenza season could be useful in planning influenza prevention and control activities. To encourage development and innovation in influenza forecasting, the Centers for Disease Control and Prevention (CDC) organized a challenge to predict the 2013–14 Unites States influenza season. Methods: Challenge contestants were asked to forecast the start, peak, and intensity of the 2013–2014 influenza season at the national level and at any or all Health and Human Services (HHS) region level(s). The challenge ran from December 1, 2013–March 27, 2014; contestants were required to submit 9 biweekly forecasts at the national level to be eligible. The selection of the winner was based on expert evaluation of the methodology used to make the prediction and the accuracy of the prediction as judged against the U.S. Outpatient Influenza-like Illness Surveillance Network (ILINet). Results: Nine teams submitted 13 forecasts for all required milestones. The first forecast was due on December 2, 2013; 3/13 forecasts received correctly predicted the start of the influenza season within one week, 1/13 predicted the peak within 1 week, 3/13 predicted the peak ILINet percentage within 1 %, and 4/13 predicted the season duration within 1 week. For the prediction due on December 19, 2013, the number of forecasts that correctly forecasted the peak week increased to 2/13, the peak percentage to 6/13, and the duration of the season to 6/13. As the season progressed, the forecasts became more stable and were closer to the season milestones. Conclusion: Forecasting has become technically feasible, but further efforts are needed to improve forecast accuracy so that policy makers can reliably use these predictions. CDC and challenge contestants plan to build upon the methods developed during this contest to improve the accuracy of influenza forecasts.

Infectious Diseases↗