In this month’s installment of the Innovation of the Month series, we explore a Georgia Tech project designed to provide information to citizens on the COVID-19 risk associated with different-sized gatherings in their local areas, called the COVID-19 Event Risk Assessment Planning Tool. MetroLab’s Ben Levine spoke with Dr. Joshua Weitz and Dr. Clio Andris, and their team from Georgia Tech on the background and development of the project.
Ben Levine: Can you describe the need for this project and who has been involved in it?
Joshua S. Weitz: I direct a research group whose aim is to understand how viruses transform human health and the fate of our planet. We had been working on COVID-19 modeling since January 2020. The origins of this project date soon thereafter, when friends and colleagues began to ask what the risk was of attending large events in early March 2020. At the time, case levels seemed low, but as we were evaluating the strength of SARS-CoV-2 and the importance of asymptomatic transmission, we were concerned that documented case reports were likely significant underestimates. As a result, I put together a simple statistical model that translated estimates of per-capita infection rates into the risk that one or more individuals had COVID-19 at events of different sizes. A graph with these risk estimates ended up going "viral" on Twitter. It was clear that there was a significant interest in translating case reports into something tangible: estimates that related risk to real-world experiences. We reached out to Prof. Andris’ group to extend our work and enable anyone to get a localized estimate of the risk of potential exposure to COVID-19 in events of different sizes.
We’ve had a great team of students and researchers involved since late Spring 2020 and sustained until now. Recent PhD graduate in Bioinformatics Dr. Aroon Chande led efforts to develop and manage the website to get it off the ground, and we thank Troy Hilley for his help on the server end. The team also included graduate students Mallory Harris (Stanford) who helped with the model and communication, and Seolha Lee (Georgia Tech) who took the lead in developing UI/UX features of the map. Dr. Stephen Beckett and Quan Nguyen from my group have been working on COVID-related issues for quite some time, and both are now helping to create maps for Europe.
Clio Andris: There was a need for a tool that puts the average user in control and gives them some agency for making their own decisions. When a user sees a map of an outbreak, they may feel helpless, and feel like a spectator in something that is unfolding. But this tool is designed to help put the user in control to some extent. For example, the statistic of 30,000 to 50,000 reported cases in the U.S. in a given day can be perceived as abstract. An easier concept to understand is the risk of an infectious person at a wedding reception or a school classroom. With the information they see, individuals can ostensibly decide to host, plan or attend an event of a given size. They can also communicate the risk to others in other areas who are planning on attending or hosting events.
Screen capture of the COVID-19 Event Risk Assessment Planning Tool for the United States, taken Oct. 17, 2020.
Levine: What factors did you consider when designing the interface for this tool? How does the design differ from other COVID-19 trackers?
Weitz: The organizing principle of this site was to focus on a particular question: What are the estimated chances that one (or more) individuals may have COVID-19 at events of different sizes? The site differs from others in focusing on that particular question and centering the views, data and dissemination of information on different ways to address and answer this question. We have felt from the outset that by doing less in this particular dashboard, we could actually help more people take an action that could make a difference in reducing the spread of COVID-19.
Aroon Chande: We wanted to build an interactive website that was still easy to use and provided more personalized risk estimates, based on locale. We chose the range of event sizes to closely align with events that people can relate to, for example a dinner party with 10 people. We encoded these as a slider to indicate the size growth, but limited the potential numbers to rounded values (10, 25, 50 … 10,000) to speed up the calculations. After releasing the U.S. maps component of the site, we saw a dramatic increase in the number of users. At the peak of the site’s popularity, we were serving over 253,000 users per day! So far, we have seen over two million unique users and served 15 million risk predictions. The website now runs on a distributed, container-based system that can support thousands of concurrent users.
We had to make certain user interface decisions, such as making buttons versus sliders. We decided to use a radio button so users would only have two options instead of having to guess their own value.
Seolha Lee: We tried to make the design easy to navigate, and we wanted to make the information easy to capture. Users can zoom and pan to any place of their choice so that they can easily find their own county — especially with the state outlines shown. We thought this would be easier than searching and clicking the name of a county using a dropdown or a search box. We used a red-orange color palette, which gives alarming signals but made it a bit transparent so it didn’t pop out too much. Also, the transparency allows users to see the names of the states and cities in the base map behind the color map, and it helps users navigate the map easily. Ultimately, we wanted users to easily share and utilize the information provided by the map tool. Thus, the tooltips that pop up when a cursor hovers over a county polygon show the name of the county and the risk level so that users can screen-capture the map while zoomed into their region.
Clio: It is similar to other tools in that many trackers tend to be using county-based maps and the dashboard-style of map updates, but this is designed to put the user at the epicenter of their own scenario. By allowing the user to choose an event size, it encourages them to envision themselves in an event — panning the crowd — and the map will tell them how many of the people there are likely to have COVID.
Levine: What data do you use to populate this model? What kind of calculations are done to generate the risk assessment?
Weitz: The website estimates the chance that one (or more) individuals have COVID-19 in events of different sizes. To do so, we first have to estimate the per-capita circulating incidence, p.To estimate p, we take the last 10 days of documented case reports divided by the local population and multiplied by an ascertainment bias. The ascertainment bias denotes the number of actual cases for each reported case; based on CDC and other seroprevalence surveys we provide two options: 5:1 and 10:1. Finally, given the per-capita circulating risk level p, we estimate the risk as 1-(1-p)^n where n is the size of the event. This simple model is simply the probability of finding one or more heads when flipping a biased coin n times where the chance of it turning up tails (no infection) is 1-p and heads (infection) is p. This binomial model assumes homogeneity of risk; nonetheless, it takes a minimal set of assumptions to move from per-capita risk to collective risk.
Lee: To populate the model, we use general population measures from the U.S. Census American Community Survey, and we use case rates from the New York Times COVID-19 API. For the map, we use general shapefiles from the U.S. Census and the typical Open Street Map basemap. Data on the ascertainment bias is still an issue we are uncertain about, but we have relied on reports from the CDC Serology Survey to estimate that five or 10 is a reasonable value to choose when creating front-facing estimations.
Levine: This tool is clearly helpful for individuals who are considering attending events — could it also be used by local or state governments? Who is your ideal use case for this project?
Weitz: The tool could be used in assessing the likely risk of exposure, to help communicate why a city or state might decide to delay an event, or change the event venue or format. Many individuals may be confused regarding the justification, and we hope the site can help communicate the quantitative rationale for such choices. In addition, the site can also be used to assess the risk of an imported case and why additional testing could help stop chains of transmission. We think such risk assessments can be part of decisions to more safely re-open public work spaces and schools, as well as assess the risk associated with indoor gatherings.
Andris: The ideal use case for this project is not to need the map anymore. But, while the virus is still in our midst, it has been useful for both individuals and for organizations. For example, we have been in contact with sports and recreation organizations who are interested in knowing about the places where they can still hold tryouts or practices. Joshua and I recently spoke with the Georgia Municipal Association and they asked a similar question, and the topics of holding a class, a town hall event and opening restaurants for indoor dining were raised. Local and state governments should use this to see whether they are willing to host or provide permits for events like political rallies, fairs, football games, parades, etc.
Levine: You are approaching COVID-19 from a computing perspective, using data to inform this model and decisions about gatherings. How has this perspective lined up with the epidemiology perspective from health professionals? Have any of your results surprised you?
Weitz: This research builds upon our team’s prior efforts to analyze Ebola virus disease dynamics and generalized work to improve understanding of the link between behavior, the characteristic of transmission at the individual scale, and dynamics at the population scale. One key idea that has become evident in managing COVID-19 is the critical importance of "super-spreading" and transmission by asymptomatic individuals. Our website has focused on what is evidently a key driver of what makes COVID-19 hard to control. Disease transmission often occurs at large gatherings, often by individuals who are unaware that they are infectious. We hope that more awareness of the risks of spreading at large events can help change behavior (e.g., mask wearing and physical distancing) to reduce spread and keep us all safer.
Screen capture of the COVID-19 Event Risk Assessment Planning Tool for Italy and the UK, taken Oct. 21, 2020.
Levine: You recently expanded to include data for the UK and Italy. Do you plan to expand to other areas? Where do you see the project going from here?
Stephen Beckett: We are actively working to expand the coverage of the COVID-19 Event Risk Assessment Planning Tool — expect more areas to be added by publication. Currently we are focused on Europe, which is experiencing a resurgence of cases, and are working with international partners to provide country-language specific versions of the site. Development takes time, but we plan to continue to add new areas to the site. We want this tool to be more useful for more people—and that means extending it to different countries — most recently to Austria, France and Switzerland as well.
Weitz: From here, we see the project expanding to help cover even more parts of the world. We want people across the globe to make more informed decisions about event planning and attending. It’s possible that in the future, we can specify more sophisticated processes such as risk of transmission or individual behavioral variables (indoor, outdoor, room size, mask-wearing, conversations, etc.), which are difficult to model at this time.
