Journal article
Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States
Proceedings of the National Academy of Sciences - PNAS, Vol.119(15), pp.e2113561119-e2113561119
04/12/2022
DOI: 10.1073/pnas.2113561119
PMCID: PMC9169655
PMID: 35394862
Abstract
SignificanceThis paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the United States. Results show high variation in accuracy between and within stand-alone models and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public-health action.
Details
- Title: Subtitle
- Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States
- Creators
- Estee Y Cramer - University of Massachusetts AmherstEvan L Ray - University of Massachusetts AmherstVelma K Lopez - Centers for Disease Control and PreventionJohannes Bracher - Heidelberg Institute for Theoretical StudiesAndrea Brennen - In-Q-TelAlvaro J Castro Rivadeneira - University of Massachusetts AmherstAaron Gerding - University of Massachusetts AmherstTilmann Gneiting - Heidelberg Institute for Theoretical StudiesKatie H House - University of Massachusetts AmherstYuxin Huang - University of Massachusetts AmherstDasuni Jayawardena - University of Massachusetts AmherstAbdul H Kanji - University of Massachusetts AmherstAyush Khandelwal - University of Massachusetts AmherstKhoa Le - University of Massachusetts AmherstAnja Mühlemann - University of BernJarad Niemi - Iowa State UniversityApurv Shah - University of Massachusetts AmherstAriane Stark - University of Massachusetts AmherstYijin Wang - University of Massachusetts AmherstNutcha Wattanachit - University of Massachusetts AmherstMartha W Zorn - University of Massachusetts AmherstYouyang Gu - Unaffiliated, New York, NY 10016.Sansiddh Jain - Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India.Nayana Bannur - Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India.Ayush Deva - Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India.Mihir Kulkarni - Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India.Srujana Merugu - Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India.Alpan Raval - Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India.Siddhant Shingi - Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India.Avtansh Tiwari - Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India.Jerome White - Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India.Neil F Abernethy - University of WashingtonSpencer Woody - The University of Texas at AustinMaytal Dahan - Computing CenterSpencer Fox - The University of Texas at AustinKelly Gaither - Computing CenterMichael Lachmann - Santa Fe InstituteLauren Ancel Meyers - The University of Texas at AustinJames G Scott - The University of Texas at AustinMauricio Tec - The University of Texas at AustinAjitesh Srivastava - University of Southern CaliforniaGlover E George - U.S. Army Engineer Research and Development CenterJeffrey C Cegan - U.S. Army Engineer Research and Development CenterIan D Dettwiller - U.S. Army Engineer Research and Development CenterWilliam P England - U.S. Army Engineer Research and Development CenterMatthew W Farthing - U.S. Army Engineer Research and Development CenterRobert H Hunter - U.S. Army Engineer Research and Development CenterBrandon Lafferty - U.S. Army Engineer Research and Development CenterIgor Linkov - U.S. Army Engineer Research and Development CenterMichael L Mayo - U.S. Army Engineer Research and Development CenterMatthew D Parno - U.S. Army Engineer Research and Development CenterMichael A Rowland - U.S. Army Engineer Research and Development CenterBenjamin D Trump - U.S. Army Engineer Research and Development CenterYanli Zhang-James - SUNY Upstate Medical UniversitySamuel Chen - SUNY Upstate Medical UniversityStephen V Faraone - SUNY Upstate Medical UniversityJonathan Hess - SUNY Upstate Medical UniversityChristopher P Morley - SUNY Upstate Medical UniversityAsif Salekin - Syracuse UniversityDongliang Wang - SUNY Upstate Medical UniversitySabrina M Corsetti - University of MichiganThomas M Baer - Trinity UniversityMarisa C Eisenberg - University of MichiganKarl Falb - University of MichiganYitao Huang - University of MichiganEmily T Martin - University of MichiganElla McCauley - University of MichiganRobert L Myers - University of MichiganTom Schwarz - University of MichiganDaniel Sheldon - University of Massachusetts AmherstGraham Casey Gibson - University of Massachusetts AmherstRose Yu - University of California San DiegoLiyao Gao - University of WashingtonYian Ma - University of California San DiegoDongxia Wu - University of California San DiegoXifeng Yan - University of California, Santa BarbaraXiaoyong Jin - University of California, Santa BarbaraYu-Xiang Wang - University of California, Santa BarbaraYangQuan Chen - University of California, MercedLihong Guo - Jilin UniversityYanting Zhao - University of Science and Technology of ChinaQuanquan Gu - University of California, Los AngelesJinghui Chen - University of California, Los AngelesLingxiao Wang - University of California, Los AngelesPan Xu - University of California, Los AngelesWeitong Zhang - University of California, Los AngelesDifan Zou - University of California, Los AngelesHannah Biegel - University of ArizonaJoceline Lega - University of ArizonaSteve McConnell - Construx, Bellevue, WA 98004.V P Nagraj - Signature Research (United States)Stephanie L Guertin - Signature Research (United States)Christopher Hulme-Lowe - Quality Assurance and Data Science, Signature Science, LLC, Austin, TX 78759.Stephen D Turner - Signature Research (United States)Yunfeng Shi - Rensselaer Polytechnic InstituteXuegang Ban - University of WashingtonRobert Walraven - Unaffiliated, Davis, CA 95616Qi-Jun Hong - Brown UniversityStanley Kong - Manhasset Secondary School, Manhasset, NY 11030Axel van de Walle - Brown UniversityJames A Turtle - Predictive Science (United States)Michal Ben-Nun - Predictive Science (United States)Steven Riley - Medical Research Council Centre for Global Infectious Disease Analysis, Department of Infectious Disease Epidemiology, School of Public Health, Imperial College, W2 1PG London, United KingdomPete Riley - Predictive Science (United States)Ugur Koyluoglu - Financial Services, Oliver Wyman, New York, NY 10036David DesRoches - Oliver Wyman Digital, Oliver Wyman, Boston, MA 02110Pedro Forli - Oliver Wyman Digital, Oliver Wyman, Sao Paolo, Brazil 04711-904Bruce Hamory - Health & Life Sciences, Oliver Wyman, Boston, MA 02110Christina Kyriakides - Oliver Wyman Digital, Oliver Wyman, New York, NY 10036Helen Leis - Health & Life Sciences, Oliver Wyman, New York, NY 10036John Milliken - Financial Services, Oliver Wyman, New York, NY 10036Michael Moloney - Financial Services, Oliver Wyman, New York, NY 10036James Morgan - Financial Services, Oliver Wyman, New York, NY 10036Ninad NirgudkarGokce Ozcan - Financial Services, Oliver Wyman, New York, NY 10036Noah Piwonka - Health & Life Sciences, Oliver Wyman, New York, NY 10036Matt RaviChris Schrader - Health & Life Sciences, Oliver Wyman, New York, NY 10036Elizabeth Shakhnovich - Health & Life Sciences, Oliver Wyman, New York, NY 10036Daniel Siegel - Financial Services, Oliver Wyman, New York, NY 10036Ryan SpatzChris Stiefeling - Financial Services, Oliver Wyman Digital, Toronto, ON, Canada M5J 0A1Barrie Wilkinson - Financial Services, Oliver Wyman, London, UK W1U 8EWAlexander Wong - Oliver Wyman Digital, Oliver Wyman, New York, NY 10036Sean Cavany - University of Notre DameGuido España - University of Notre DameSean Moore - University of Notre DameRachel Oidtman - University of ChicagoAlex Perkins - University of Notre DameDavid Kraus - Masaryk UniversityAndrea Kraus - Masaryk UniversityZhifeng Gao - Microsoft, Redmond, WA 98029Jiang Bian - Microsoft, Redmond, WA 98029Wei Cao - Microsoft, Redmond, WA 98029Juan Lavista Ferres - Microsoft, Redmond, WA 98029Chaozhuo Li - Microsoft (Israel)Tie-Yan Liu - Microsoft, Redmond, WA 98029Xing Xie - Microsoft, Redmond, WA 98029Shun Zhang - Microsoft, Redmond, WA 98029Shun Zheng - Microsoft, Redmond, WA 98029Alessandro Vespignani - Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA 02115Matteo Chinazzi - Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA 02115Jessica T Davis - Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA 02115Kunpeng Mu - Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA 02115Ana Pastore Y Piontti - Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA 02115Xinyue Xiong - Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA 02115Andrew Zheng - Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139Jackie Baek - Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139Vivek Farias - Massachusetts Institute of TechnologyAndreea Georgescu - Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139Retsef Levi - Massachusetts Institute of TechnologyDeeksha Sinha - Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139Joshua Wilde - Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139Georgia Perakis - Massachusetts Institute of TechnologyMohammed Amine Bennouna - Massachusetts Institute of TechnologyDavid Nze-Ndong - Massachusetts Institute of TechnologyDivya Singhvi - Technology, Operations and Statistics (TOPS) group, Stern School of Business, New York University, New York, NY 10012Ioannis Spantidakis - Massachusetts Institute of TechnologyLeann Thayaparan - Massachusetts Institute of TechnologyAsterios Tsiourvas - Massachusetts Institute of TechnologyArnab Sarker - Massachusetts Institute of TechnologyAli Jadbabaie - Massachusetts Institute of TechnologyDevavrat Shah - Massachusetts Institute of TechnologyNicolas Della Penna - Massachusetts Institute of TechnologyLeo A Celi - Massachusetts Institute of TechnologySaketh Sundar - River Hill High School, Clarksville, MD 21029Russ Wolfinger - JMP Life Sciences, SAS Institute Inc, Cary, NC 27513Dave Osthus - Los Alamos National LaboratoryLauren Castro - Los Alamos National LaboratoryGeoffrey Fairchild - Los Alamos National LaboratoryIsaac Michaud - Los Alamos National LaboratoryDean Karlen - Physical Sciences DivisionMatt Kinsey - Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723Luke C Mullany - Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723Kaitlin Rainwater-Lovett - Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723Lauren Shin - Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723Katharine Tallaksen - Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723Shelby Wilson - Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723Elizabeth C Lee - Johns Hopkins UniversityJuan Dent - Johns Hopkins UniversityKyra H Grantz - Johns Hopkins UniversityAlison L Hill - Johns Hopkins UniversityJoshua Kaminsky - Johns Hopkins UniversityKathryn Kaminsky - Unaffiliated, Baltimore, MD 21205Lindsay T Keegan - University of UtahStephen A Lauer - Johns Hopkins UniversityJoseph C Lemaitre - École Polytechnique Fédérale de LausanneJustin Lessler - Johns Hopkins UniversityHannah R Meredith - Johns Hopkins UniversityJavier Perez-Saez - Johns Hopkins UniversitySam Shah - Unaffiliated, San Francisco, CA 94122Claire P Smith - Johns Hopkins UniversityShaun A Truelove - Johns Hopkins UniversityJosh Wills - Unaffiliated, San Francisco, CA 94122Maximilian Marshall - Johns Hopkins UniversityLauren Gardner - Johns Hopkins UniversityKristen Nixon - Johns Hopkins UniversityJohn C Burant - Unaffiliated, Amsterdam, The NetherlandsLily Wang - Iowa State UniversityLei Gao - Iowa State UniversityZhiling Gu - Iowa State UniversityMyungjin Kim - Iowa State UniversityXinyi Li - Clemson UniversityGuannan Wang - William & MaryYueying Wang - Iowa State UniversityShan Yu - University of VirginiaRobert C Reiner - University of WashingtonRyan Barber - University of WashingtonEmmanuela Gakidou - University of WashingtonSimon I Hay - University of WashingtonSteve Lim - University of WashingtonChris Murray - University of WashingtonDavid Pigott - University of WashingtonHeidi L Gurung - AIRPrasith Baccam - AIRSteven A Stage - Emerging Technologies, IEM, Inc., Baton Rouge, LA 70809Bradley T Suchoski - AIRB Aditya Prakash - Georgia Institute of TechnologyBijaya Adhikari - University of IowaJiaming Cui - Georgia Institute of TechnologyAlexander Rodríguez - Georgia Institute of TechnologyAnika Tabassum - Virginia TechJiajia Xie - Georgia Institute of TechnologyPinar Keskinocak - Georgia Institute of TechnologyJohn Asplund - Advanced Data Analytics, Metron, Inc., Reston, VA 20190Arden Baxter - Georgia Institute of TechnologyBuse Eylul Oruc - Georgia Institute of TechnologyNicoleta Serban - Georgia Institute of TechnologySercan O Arik - Google Cloud, Sunnyvale, CA 94089Mike Dusenberry - Google Cloud, Sunnyvale, CA 94089Arkady Epshteyn - Google Cloud, Sunnyvale, CA 94089Elli Kanal - Google Cloud, Sunnyvale, CA 94089Long T Le - Google Cloud, Sunnyvale, CA 94089Chun-Liang Li - Google Cloud, Sunnyvale, CA 94089Tomas Pfister - Google Cloud, Sunnyvale, CA 94089Dario Sava - Google Cloud, Sunnyvale, CA 94089Rajarishi Sinha - Google Cloud, Sunnyvale, CA 94089Thomas Tsai - Harvard UniversityNate Yoder - Google Cloud, Sunnyvale, CA 94089Jinsung Yoon - Google Cloud, Sunnyvale, CA 94089Leyou Zhang - Google Cloud, Sunnyvale, CA 94089Sam Abbott - London School of Hygiene & Tropical MedicineNikos I Bosse - London School of Hygiene & Tropical MedicineSebastian Funk - London School of Hygiene & Tropical MedicineJoel Hellewell - London School of Hygiene & Tropical MedicineSophie R Meakin - London School of Hygiene & Tropical MedicineKatharine Sherratt - London School of Hygiene & Tropical MedicineMingyuan Zhou - The University of Texas at AustinRahi Kalantari - The University of Texas at AustinTeresa K Yamana - Columbia UniversitySen Pei - Columbia UniversityJeffrey Shaman - Columbia UniversityMichael L Li - Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139Dimitris Bertsimas - Massachusetts Institute of TechnologyOmar Skali Lami - Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139Saksham Soni - Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139Hamza Tazi Bouardi - Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139Turgay Ayer - Emory UniversityMadeline Adee - Massachusetts General HospitalJagpreet Chhatwal - Massachusetts General HospitalOzden O Dalgic - Health Economic Modeling, Value Analytics Labs, 34776 İstanbul, TurkeyMary A Ladd - Massachusetts General HospitalBenjamin P Linas - Boston University School of MedicinePeter Mueller - Massachusetts General HospitalJade Xiao - Georgia Institute of TechnologyYuanjia Wang - Columbia UniversityQinxia Wang - Columbia UniversityShanghong Xie - Columbia UniversityDonglin Zeng - University of North Carolina at Chapel HillAlden Green - Department of Statistics, Carnegie Mellon University, Pittsburgh, PA 15213Jacob Bien - Marshall School of Business, Department of Data Sciences and Operations (DSO), University of Southern California, Los Angeles, CA 90089Logan Brooks - Department of Statistics, Carnegie Mellon University, Pittsburgh, PA 15213Addison J Hu - Department of Statistics, Carnegie Mellon University, Pittsburgh, PA 15213Maria Jahja - Department of Statistics, Carnegie Mellon University, Pittsburgh, PA 15213Daniel McDonald - University of British ColumbiaBalasubramanian Narasimhan - Stanford UniversityCollin Politsch - Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA 15213Samyak Rajanala - Stanford UniversityAaron Rumack - Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA 15213Noah Simon - University of WashingtonRyan J Tibshirani - Department of Statistics, Carnegie Mellon University, Pittsburgh, PA 15213Rob Tibshirani - Stanford UniversityValerie Ventura - Department of Statistics, Carnegie Mellon University, Pittsburgh, PA 15213Larry Wasserman - Department of Statistics, Carnegie Mellon University, Pittsburgh, PA 15213Eamon B O'Dea - Odum School of Ecology, University of Georgia, Athens, GA 30602John M Drake - Odum School of Ecology, University of Georgia, Athens, GA 30602Robert Pagano - Unaffiliated, Tucson, AZ 85704Quoc T Tran - Catalog Data Science, Walmart Inc., Sunnyvale, CA 94085Lam Si Tung Ho - Dalhousie UniversityHuong Huynh - Virtual Power System Inc, Milpitas, CA 95035Jo W Walker - COVID-19 Response, Centers for Disease Control and Prevention; Atlanta, GA 30333Rachel B Slayton - COVID-19 Response, Centers for Disease Control and Prevention; Atlanta, GA 30333Michael A Johansson - COVID-19 Response, Centers for Disease Control and Prevention; Atlanta, GA 30333Matthew Biggerstaff - COVID-19 Response, Centers for Disease Control and Prevention; Atlanta, GA 30333Nicholas G Reich - University of Massachusetts Amherst
- Resource Type
- Journal article
- Publication Details
- Proceedings of the National Academy of Sciences - PNAS, Vol.119(15), pp.e2113561119-e2113561119
- DOI
- 10.1073/pnas.2113561119
- PMID
- 35394862
- PMCID
- PMC9169655
- NLM abbreviation
- Proc Natl Acad Sci U S A
- ISSN
- 0027-8424
- eISSN
- 1091-6490
- Publisher
- National Academy of Sciences
- Grant note
- R01 EB017205 / NIBIB NIH HHS
- Language
- English
- Date published
- 04/12/2022
- Academic Unit
- Computer Science
- Record Identifier
- 9984259431902771
Metrics
17 Record Views