Multi-source transformer forecasting
A transformer that fuses case, death, and social media sentiment data for county-level COVID-19 forecasting, evaluated retrospectively across 46 South Carolina counties and three Omicron waves.
Wu J, Tanim S, Woo M, Ahammed T, Bleichrodt AM, Rennert L. Epidemics 53 (2025) 100865. Open access (CC BY 4.0).
Problem
Epidemic forecasts often depend only on delayed indicators such as confirmed cases. Prior work using social media signals focused on detecting outbreaks early, and often only in counties of more than a million people, which can miss outbreaks in less populous areas. The study asked whether a transformer that fuses epidemiological records with county-level social media sentiment could forecast cases and deaths for all 46 counties of one state.
Data
- Epidemiological records, January 2020 to March 2023: county cases and deaths from The New York Times, and positive cases, hospitalization rates, and testing data from the CDC. Targets are the 7-day averages of cases and deaths.
- Demographics: county and health service area populations.
- Social media: posts on X (Twitter) about COVID-19 in South Carolina, collected through Sprinklr with geolocation and keyword filters, mapped to counties, and reduced to three sentiment channels (positive, negative, neutral) from Sprinklr's labels.
Model
- Encoder-only transformer, trained from scratch: 512-dimensional input embeddings, 4 encoder layers with 8 attention heads, sinusoidal positional encoding, and a linear projection to the next k values. The decoder was dropped because it added overfitting risk and instability in multi-step forecasting.
- Sentiment fusion: the three sentiment channels pass through a 1 × 1 convolution and a fully connected layer, and are added to the embeddings of the other features.
- Direct multi-step forecasting: a separate model for each horizon (up to 7 days ahead, or up to 5 weeks ahead) instead of feeding predictions back in, which avoids accumulating errors.
- One model for all counties, trained on randomly sampled counties, start dates, and window lengths (look-back of up to 16 days or 5 weeks), so that it learns from many phases of the pandemic rather than one.
Evaluation design
- Time-ordered three-way split: training January 2020 to September 2021; validation October to November 2021, used only for tuning and early stopping; test December 2021 to March 2023, used once at the end.
- Test periods matched to the Omicron waves: B.1.1.529 (December 2021 to March 2022), BA.4/BA.5 (May to October 2022), and BQ.1/XBB.1.5 (December 2022 to February 2023).
- Metrics: county-level percentage agreement between predicted and observed values (median and interquartile range across counties), RMSE, and MAE.
- Comparisons: a persistence baseline (last observation carried forward), a per-county auto-ARIMA, and the same model without social media data.
Results
Median county agreement across the three waves:
| Target | 1 week ahead | 5 weeks ahead |
|---|---|---|
| Cases | 74.0–82.6% | 68.7–79.6% |
| Deaths | 83.2–86.3% | 84.3–87.2% |
Against the baselines, as reported in the paper's comparison of longer- and shorter-term forecasts:
| Horizon, metric | This model | Persistence | ARIMA |
|---|---|---|---|
| 5 weeks, agreement | 81.9% | 61.2% | 59.9% |
| 5 weeks, RMSE | 35.1 | 98.4 | 106.9 |
| 5 weeks, MAE | 23.0 | 72.1 | 77.6 |
| 7 days, agreement | 74.5% | 71.6% | 55.9% |
The advantage over persistence is large at five weeks and small at seven days, where carrying the last value forward is a strong baseline. Accuracy falls as the horizon grows, and predicted peaks consistently lag the observed ones.
What social media data added
Change in median case agreement from adding sentiment data, by wave (range over horizons):
| Wave | 1–7 days ahead | 1–5 weeks ahead |
|---|---|---|
| 1 (B.1.1.529) | +4.94 to +10.50% | +9.04 to +14.82% |
| 2 (BA.4/BA.5) | −4.22 to +4.06% | −2.69 to +10.09% |
| 3 (BQ.1/XBB.1.5) | −3.57 to +3.53% | −0.49 to +7.52% |
The gain was clear in the first Omicron wave, a period of rapid change and heavy public discussion, and mixed afterwards. Overall the paper describes it as a mild to moderate improvement.
Limitations
- Retrospective only. The evaluation assumes complete, timely data. Live data streams can be delayed, revised, or stopped: The New York Times ended its COVID-19 reporting in March 2023, which ended validation with that source.
- Point forecasts. The model gives no prediction intervals, so it was not evaluated probabilistically, which is the standard for epidemiological forecasting.
- One state. Performance elsewhere is unproven; a new region would need local retraining and validation.
- Social media is an indirect signal. News coverage and platform changes move it independently of disease spread, so its value can drift over time.
What carried forward
Later work addresses the first two limitations. Evaluating on data as it was published at the time, rather than on later revised values, is the basis of ObsAwareData, which keeps every release and a reliability flag on each value, and of the planned real-time benchmark in resplens, which is to report probabilistic scores.