Forecasting COVID-19, Influenza and RSV hospitalisations over winter 2023/24 in England
Data Analytics and Surveillance Group, UK Health Security Agency, London, UK
School of Public Health, Imperial College London, London, UK
*Corresponding author: jonathon.mellor@ukhsa.gov.ukAbstract
Seasonal respiratory viruses cause substantial pressure on healthcare systems, particularly over winter. System managers can mitigate the impact on patient care when they anticipate hospital admissions due to these viruses. Hospitalisation forecasts were used widely during the SARS-CoV-2 pandemic. Now, resurgent seasonal respiratory pathogens add complexity to system planning. We describe how a suite of forecasts for respiratory pathogens, embedded in national and regional decision-making structures, were used to mitigate the impact on hospital systems and patient care.
We developed forecasting models predicting hospital admissions and bed occupancy two weeks ahead for COVID-19, influenza, and RSV in England over winter 2023/24. Bed occupancy forecasts were informed by the ensemble admissions models. Forecasts were delivered in real-time at multiple scales. The use of sample-based forecasting allowed for effective reconciliation and trend interpretation.
Admission forecasts, particularly RSV and influenza, showed high skill at regional levels. Bed occupancy forecasts had well-calibrated coverage, owing to informative admissions forecasts and slower moving trends. National admissions forecasts had mean absolute percentage errors of 27.3%, 30.9% and 15.7% for COVID-19, influenza, and RSV respectively, with corresponding 90% coverages of 0.439, 0.807 and 0.779.
These real-time winter infectious disease forecasts produced by the UK Health Security Agency for healthcare system managers played an informative role in mitigating seasonal pressures. The models were delivered regularly and shared widely across the system to key users. This was achieved by producing reliable, fast, and epidemiologically informed ensembles of models. Though, a higher diversity of model approaches could have improved forecast accuracy.
Article notes
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding
1.Introduction
In England, each year respiratory pathogens circulating in the community cause strain on the healthcare system, particularly SARS-CoV-2, influenza, and Respiratory Syncytial Virus (RSV). In the 2023/24 winter season there were 96.6, 76.9 and 37.8 cumulative admissions per 100,000 people in England with COVID-19, influenza, and RSV respectively [1] [2]. Public health and healthcare leaders can better use scarce resources when they can predict hospital admissions and bed occupancy, reallocating staff and beds to elective care as emergency admissions decline.
Forecasting is widely used to give quantitative assessments on future events, harnessing historic trends and modelled assumptions. However, infectious disease dynamics are challenging to confidently understand in real-time due to their complexity and partial observation. When used well, numerical forecasts can help inform healthcare planning to allocate system resources more effectively, leading to improved health outcomes. Within England the UK Health Security Agency (UKHSA) is responsible for the prevention, preparation, and response to infectious diseases. The National Health Service England (NHSE) provides healthcare provision to the population, and both government organisations are overseen by the Department of Health and Social Care.
Infectious disease forecasts have been produced for influenza over many years in the United States by the Centre for Disease Control (CDC) [3] [4]. Forecasting is not limited to influenza, with challenges covering West Nile Virus [5], Ebola [6], and Chikungunya [7], with collaborations between government and academia driving improvements in the field [8] [9]. These methods and initiatives were stepped up during the SARS-CoV-2 pandemic responding to increased demand [10] [11], used in communications by the US CDC to the public.
The SARS-CoV-2 pandemic disrupted seasonal patterns of many respiratory viruses, driven by changing population mixing dynamics caused by control interventions [12], notably limiting influenza and RSV transmission. As population mixing returned to pre-pandemic levels these diseases re-emerged in winter 2022/23 [13]. Rising emergency hospital admissions due to influenza, COVID-19, and RSV place direct resource pressure on hospitals [14]. These admissions take up in-patient beds, and the increased demand on hospitals reduces the planned and non-emergency care available. Therefore, the accurate prediction of infectious disease pressures helps to manage the wider healthcare system.
A range of metrics are commonly used to measure and understand seasonal respiratory disease waves, each tackling different policy questions: peak magnitude, peak timing, cumulative incidence, and incidence over time [4]. Comparisons between rapid collections and retrospective records give us confidence in our choice of metrics indicating suitability for real-time modelling [15]. In addition, we focus on test-positive diagnosis rather than syndrome to ensure modelling is specific to a pathogen, allowing us to model diseases separately [16].
Over the winter 2023/24 season UKHSA delivered a suite of forecasting models to estimate hospital admissions and bed occupancy trends of COVID-19, influenza, and RSV waves. This breadth of pathogens and metrics modelled was key to the suite’s utility. The forecasts were used directly to inform national policy discussions and integrated into regional level decision making. In this work we outline our approach in delivering this forecasting suite, key considerations, and describe our real-time results.
2.Methods
2.1Data sources
The National Health Service (NHS) is a publicly funded healthcare system covering England, with data collection consistent across hospitals [17]. NHS England geographic structures are given in Supplementary Figure 1. Individuals presenting severe symptoms of an infectious disease are often tested in secondary care settings, with diagnostic tests reported to UKHSA via the Second-Generation Surveillance System (SGSS). There is a high ascertainment and coverage of SARS-CoV-2, influenza, and RSV testing in hospitals for differential diagnosis. However, this ascertainment can vary with demographics such as age [18] and across primary care, secondary care, and community settings [19]. More broadly, all-cause hospital admissions can be predicted by seasonal patterns and environmental effects [20]. However, presentations due to infectious disease are less predictable, particularly when seasonal transmission patterns are disrupted. Test-positive hospitalisations; admissions and bed occupancy, are key metrics for the pressures sustained by hospitals due to infectious disease. Forecast target definitions are provided (Table 1), highlighting their differences.
2.2Forecast Models
Statistical and ensemble rationale
Our modelling suite focusses on statistical forecasts, aligning with the evidence of their strong performance in short-term forecasting tasks, relative to strictly mechanistic approaches [21] [22]. Alongside their predictive performance, this class of models has fast runtimes, are largely disease agnostic, and do not require substantial historic data. Furthermore, we developed model ensembles, which capture uncertainty in possible trends [23], with evidence of improved performance [24] [25]. Model ensembles also reduce our reliance on individual models, improving operational resilience.
Ensemble inclusion criteria
In a M-open model combination, for possible candidate models M, we ideally have a breadth of modelling approaches to produce combination estimates [26]. However, practical considerations are critical in ensemble design. Across pathogens and metrics, there is a single-day turnaround from modelling to dissemination, highlighting the importance of speed and reliability. Candidate models are selected if the model: Criteria 1 makes the production of all forecasts feasible, allowing for error flagging, model reruns, and thorough assurance of results. While models can be run in parallel, there are limitations in how many individuals are available to run the process, make fixes, and quality check results. Criteria 2 ensures desirable qualities for ensembling and coherence of prediction aggregations, such as from local to national levels. Criteria 3 is a holistic evaluation of model performance and utility. We do not select models based on scoring rules alone, but instead consider the wider properties of a model (reliability, ease-of-use, difference from existing models) and discussion with surveillance experts.
- has a total runtime (fitting, inference, post-processing) across all locations for a disease in fewer than 15 minutes.
- produces prediction samples (such as posterior samples).
- improves forecasting capability.
Specific model classes
A variety of statistical models were used to forecast trends, each with different underlying assumptions. Each model is structured separately for each disease, adapted to best reflect the surveillance data being modelled, with key hyperparameters retuned regularly over the season. The primary model class used across the pathogens was a Generalised Additive Model (GAM), which uses the semi-mechanistic assumption of a recent growth-rate extrapolated forward in time [27] via the mgcv and gratia packages [28] [29]. This model has different variations across the diseases, using different hierarchical components, geographies, and for the RSV model a structure pooling trends across adjacent age groups. To improve model performance, we leverage probabilistic catchment areas to define a denominator population for hospital admissions [30]. Secondly, we use state-space based models, primarily with an Error Trend Seasonality (ETS) structure fit to each individual geographic location’s time series via the fable package [31]. Lastly, syndromic surveillance is fed into regression models to predict expected future admissions given current leading indicators levels [32]. Models ran each week are first inspected by the modelling team (applying expert judgement), then candidate models are ensembled. Each model developed is given by disease in Supplementary Table 1.
For the COVID-19 and influenza admissions targets we developed a secondary forecast target requested by users, beds occupied with test-positive patients. This is generated by fitting a convolution between the admissions and occupancy time series, giving an estimate for time-to-discharge, translating the admissions forecast into an occupancy forecast.
To compare models and tune parameters we use probabilistic scoring methods, primarily the weighted interval score, empirical coverage, and bias, calculated using the scoringutils [33] R package. 90% and 50% prediction intervals are generated, with the 90% and median value are communicated to users, alongside probabilistic statements assessing trends. While quantitative scoring is a key component in defining and improving our ensemble, expert judgement in the modelling team is necessary to exclude models in a given week.
Benefits of the stacking ensemble approach
Variations on the unweighted average quantile approach to ensembling are widely used in epidemic forecasting [34] [25]. Instead, we preserve individual model sample predictions [35], then perform an unweighted quantile summary of draws across all models. This allows us to capture uncertainty in a granular way when summarising predictions and avoid relying on averaging or aggregating point estimates.
Though this prediction sample approach requires higher data volume than point estimates, as models are run within one team this has a limited impact on operations. By using a prediction sample stacking approach, we can produce granular forecasts at local levels then aggregate the sample draws to higher geographies, allowing for simple reconciliation across space, time, and other quantities of interest (such as age in the RSV model) [36]. It is important to our users that we produce national forecasts (for ministers), low geographies (for health protection teams), and their coherence is important for credibility.
Lastly, the use of prediction samples allows us to assign probabilities to different interpretable trend categories (stable, increase and decrease) using thresholds agreed with disease and operations experts, achievable at all geographies. We consider a change over two weeks of <20% to be stable, a positive change >20% as an increase and a negative change >20% as a decrease, similar to other experimental approaches worldwide [37].
3.Results
In a time of disrupted and uncertain seasonal patterns caused by the SARS-CoV-2 pandemic we chose to forecast our metrics over a forecast horizon window, showing how the metric is expected to develop. As a trade-off between long forecast horizons with high uncertainty and short horizons with lower utility, we chose a consistent 14-day horizon across all models. This provides enough foresight to be useful, with meaningfully small uncertainty, agreed with end users of the forecasts.
3.1Season overview and targets
Each disease is monitored differently across the healthcare system, with different levels of granularity and quality available. The forecast targets selected for each disease and metric are given in Table 1. A key consideration in the choice of target data is its latency, traded off against its accuracy. For operational efficiency, forecasts for each disease were not delivered across every winter week. Forecasts were prioritised based on the trends observed, stopped early if needed, new models incorporated, and further metrics included as the season progressed. The timeline of metrics forecasted over the season are given in Figure 1.
Our confidence in each data stream, and what is interesting to the users of our forecast varies by disease, and therefore the spatial resolution we model is not consistent across diseases. COVID-19 admissions have the highest quality of reporting, in part due to their high priority during the pandemic, though age information is no longer captured. Alternatively, for RSV, as individual data are processed by UKHSA, stratifications by age are possible allowing for models that incorporate demographic effects.
3.2Influenza
The seasonal influenza epidemic in 2023/24 was characterized by a small peak before the New Year, and a second larger peak after (Figure 2). Observed 14-day admissions and occupancy values were within the forecasted 90% prediction intervals in 15/19 and 10/11 weeks respectively, though the admissions forecast performed poorly between peaks (Figure 2). For influenza, the GAM was included in the ensemble across all weeks, the ETS model incorporated later in the season, and leading indicator models used intermittently.
3.3COVID-19
Unlike influenza, there were COVID-19 admissions at non-zero levels at the start of winter, with multiple peaks and troughs observed in the admissions and occupancy trends (Figure 3). The main peak occurred near the New Year. The COVID-19 forecasts have variation in their accuracy, with forecasts generally struggling in growth/decline phases, where the models are slow to adapt to changing trends (Figure 4). Of the five available models for the ensemble, none were used across all weeks. The GAM was used most frequently, with sporadic use of leading indicator models, and the ETS model was regularly deployed towards the end of the season.
4.4RSV
Relative to influenza and COVID-19, the peak in the RSV epidemic was earlier in the season, occurring in November (Figure 4). All but two 14-day forecasted 90% prediction intervals contain the observed values, capturing the trend in the epidemic well across the wave (Figure 4). One single model, the GAM with age and regional stratifications, was used to forecast this year.
4.5Forecast performance
Quantitative evaluation of forecast accuracy is critical to the communication and improvement of forecasts. Measures of performance at a range of breakdowns show varying performance across diseases, metrics, and geographies (Table 2). Across 90% and 50% coverage scores the regional forecasts were better calibrated than their national aggregations. The occupancy forecasts for COVID-19 and influenza had higher performing central estimates than the corresponding admissions forecasts, a result of the slower evolving trend dependent on past admissions.
The forecast performances can also be explored relative to the winter epidemic wave peak (Figure 5). For the influenza admissions, occupancy, and COVID-19 admissions there is a notable drop in coverage performance following the epidemic peak, indicating models may struggle to be calibrated at this point. Notably, as with the results in Table 2, the regional coverage is consistently better than at national levels, particularly for the COVID-19 models, even across epidemic phases. Further scoring metrics, such as the coverage deviation and bias are explored in Supplementary Tables 2 & 3.
4.Discussion
In this manuscript we describe the approach taken by modellers at UKHSA to deliver a suite of forecasts across three priority diseases in the 2023/24 winter season. Forecasts harnessed the unified data collection of the health service in England, giving high coverage of hospitals, low latency, and granular spatio-temporal resolution. Some forecasts were particularly accurate over the season, such as the RSV admissions results (Figure 4). However, modelling for COVID-19 admissions were particularly challenging this season (Figure 2). The forecasts were widely disseminated within the health system, with modellers and operations colleagues collaborating closely. The resulting forecasts had high utility at both national and regional geographies, aiding decision makers in their resource allocation decisions.
Forecast users ranged from senior ministers responsible for the nation’s health, to regional health protection teams, with a variety of use-cases in between (Figure 6). National forecasts were shared each week with the Department of Health and Social Care to support oversight by the Secretary of State and ministers. This aided national-level decisions on system performance and operating policy over the winter, along with other analyses. At a sub-national level, forecasts were shared at weekly regional operational meetings held by NHS England. This gave foresight for regional decision-makers on hospital capacity, operational pressure [38], and coordination of mutual aid between hospitals and regions. Forecasts were also used by regional health protection teams to anticipate pressure on public health response.
Developing a modelling suite in-house allowed UKHSA modellers to leverage existing relationships between disease experts and data collectors to develop useful forecasts. Granular, unified data collection enabled geography-specific forecasts to be created, directly supporting local operations, strengthening relationships between users and forecasters. By developing the forecasting suite within a single team, it was straightforward to create new targets and experiment with new approaches, which will be necessary in future outbreaks. By focusing on statistical time-series methods, the suite can be scaled to new diseases and metrics, without the challenging parameterisation of more mechanistic approaches. A range of model types can be used with the sample-based ensembling, giving strong reconciliation across geographic scales, and allowing for trend categorisation that aids decision making.
The modelling suite was developed iteratively over the past three years. It began in 2021 as a single COVID-19 model, then for winter 2022/23 evolved into a COVID-19 ensemble and a new influenza model. Most recently, in 2023/24 we created model ensembles for both COVID-19 and influenza, with an RSV model added. This build-up of models over time allowed adaption to evolving user needs, experimentation with new approaches, and streamlining processes in quieter periods.
The choice of forecasting target underpins the utility of the modelling, particularly the forecast horizon. Statistical models have been shown to excel at short horizons, relative to longer time periods [22]. Our work has a short-term focus, with a 14 day-look-ahead, providing more reasonable uncertainty in trends compared to other methods. Though, there is less room for decision makers to plan as the horizon shortens. We have found, while short, the 2-week horizon is highly useful for stakeholders, with meaningful uncertainty. Scenario modelling over longer term horizons has a different purpose to the immediate decision making of short-term forecasting, where academic collaborations have excelled [39].
The natural extensions to this modelling suite are clear: improvements in forecast accuracy, extending the forecast horizon, and exploration of other disease targets and further granularity. To further improve situational awareness, combinations of nowcasting and forecasting should be considered to harness inherently lagged data. The current suite of models is primarily statistical. Deploying models with more mechanistic approaches, including further disease dynamics assumptions, and incorporating historical trends are further avenues for exploration [40]. Further diseases should be considered for forecasting, creating the capability for responsive modelling work, though this necessitates improved timeliness of data collection, which currently exists only for influenza and COVID-19 in England. Increased sharing of data and code would benefit methods development and assurance, though we must be careful this does not limit modellers’ agility to deliver in response periods, and flexibility to try new targets and metrics.
The choice of modelling in-house only, largely due to data sharing challenges, limits the public availability of forecast information in real-time. The work being closed source, rather than code existing in the public domain, limits the scrutiny and collaboration available for the project. Furthermore, by focusing efforts within a single team there is a limit to time available for novel model development. This leaves expertise, primarily academic, both within England and internationally, that is not necessarily being leveraged to inform predictions – highlighting the importance of government and academic collaboration. The ensemble is limited by a lack of mechanistic dynamic components, such as susceptible population depletion, which may help with performance at epidemic turning points.
Over the 2023/24 winter season, we delivered a forecasting suite for respiratory diseases within secondary care in England with a range of statistical and ensemble models. These model outputs were used widely across the health system for situational awareness and supported decision making at national and local levels. We have shown that having an internal capability for real-time modelling of infectious diseases supports the delivery of effective public health.
Supporting information
Acknowledgements
We would like to specifically thank members of the Infectious Disease Modelling team for their contributions to the operational delivery of the winter forecasts over the season:
- –William Ferguson
- –Jack Kennedy
- –Emilie Finch
- –Oliver Polhill
- –Chetan Chauhan-Sims
- –Adrian Pritchard
- –Rachel Christie
In addition, we would like to thank the analysts within the NHS England Urgent and Emergency Care teams for their support, as well as colleagues within the NHS collecting and reporting the data used in this work.
Ethical Approval
UKHSA have an exemption under regulation 3 of section 251 of the National Health Service Act (2006) to allow identifiable patient information to be processed to diagnose, control, prevent, or recognise trends in, communicable diseases and other risks to public health.
Conflict of Interest
The authors have declared that no competing interests exist.
Data Availability Statement
UKHSA⍰operates a robust governance process for applying to access protected data that considers:
- the benefits and risks of how the data will be used
- compliance with policy, regulatory and ethical obligations
- data minimisation
- how the confidentiality, integrity, and availability will be maintained
- retention, archival, and disposal requirements
- best practice for protecting data, including the application of “privacy by design and by default”, emerging privacy conserving technologies and contractual controls
Access to protected data is always strictly controlled using legally binding data sharing contracts.
UKHSA⍰welcomes data applications from organisations looking to use protected data for public health purposes.
To request an application pack or discuss a request for UKHSA data you would like to submit, contact DataAccess@ukhsa.gov.uk.