How to spot good Covid-19 vaccine trials results when you see them

Monday marked the first announcement of preliminary results of Phase 3 trials of Covid-19 vaccines, and we expect many more to follow.

These results are long awaited: Mass vaccination will give us the most promising path out of the pandemic — but only if people trust the results enough to get the vaccine. A well-informed public is among the most crucial elements of a successful vaccination program.

It doesn’t take an epidemiologist to thoughtfully evaluate vaccine trial results. Answers to just a handful of questions can help anyone compare their strengths and weaknesses, distinguish the promising from the inadequate, and know when more information is needed to figure out which is which.

Did the vaccine reduce symptomatic infections? No vaccine is 100% protective. The best of the bunch, like vaccines against the measles and hepatitis A, reduce infections in about 95% of people. Yet many other vaccines, such as those for influenza and typhoid, are closer to a coin flip — getting vaccinated reduces the risk of infection by about half.

What is the target for Covid-19 vaccines? The FDA has set an efficacy threshold of a 50% reduction in symptomatic infections for consideration of licensure (total and severe infections are harder to measure — more on that later). But experts would like to see something closer to a 60% to 70% reduction, which would bring us closer to approaching herd immunity and potentially allow us to start relaxing physical distancing measures. This is one reason why the 90% reduction reported this week by Pfizer and BioNTech in the first trial results are particularly exciting.

Who participated in the study? Three groups are key: people older than 65; underrepresented racial and ethnic groups, particularly Black and Latino populations; and individuals with medical conditions such as kidney disease, diabetes, or obesity. Study after study after study has shown that these groups suffer the greatest burden of Covid-19-related complications and death. So a vaccine that reduces disease severity in young, healthy white people but has a little effect in these higher-risk groups might have considerably less public health impact.

Pharmaceutical companies have been called upon for years to increase diversity in clinical trials, particularly among historically marginalized and exploited groups. Although the Phase 3 trials for Covid-19 vaccines are not large enough to separately show benefit or harm in each of these groups, inclusion of equitable numbers will be crucial for broad public trust in trial results.

What do the results reveal about severe infections? On first glance, this might seem like a rehash of the first point, about whether the vaccine reduced symptomatic infections. But the distinction is very important. A vaccine that only moderately prevents infections but that strongly reduces hospitalizations or deaths could still be a winning candidate. It would convert Covid-19 into something akin to the common cold — an annoyance, but not a life-threatening infection. Reducing disease severity may also make people less likely to transmit the virus to others, another win.

Because severe Covid-19 infections are relatively uncommon — occurring more often in the elderly, less so in young, healthy people — showing a large benefit in reducing disease severity won’t be easy. For example, the clinical trial plans for the Pfizer/BioNTech vaccine calls for enrolling approximately 43,000 people to detect just 160 symptomatic infections when the final results are released (there were 96 detected symptomatic infections at the time of the preliminary analysis). No data on severe infections have been reported so far, but we would expect only about two dozen of the 160 infected to be hospitalized and a few to die, leaving little chance to show that the vaccine prevents severe complications. Nonetheless, even a modest effect on Covid-19 severity would be welcome, especially if it occurs among the vulnerable groups discussed earlier.

Is it safe? Since vaccines are mainly given to healthy people — and in this case, potentially to billions of them — understanding the safety of a Covid-19 vaccine is essential. Some good news is that the vaccine candidates under consideration are unlikely to have common severe side effects such as a severe allergic reaction or rash requiring hospitalization. They have completed Phase 2 trials that demonstrated safety in hundreds of people. Any vaccine that caused severe side effects in more than 10% of the Phase 2 participants would not have moved ahead into these larger trials.

Mild side effects should not raise too much concern. Most vaccines, including many of the Covid-19 candidates, include additives called adjuvants that intentionally “rev up” the immune system to respond to the vaccine. These can frequently cause arm pain for a few days or flu-like symptoms. Such side effects are a worthwhile trade-off if in exchange for preventing Covid-19 and allowing us to safely reopen society.

But severe complications — especially those that lead to hospitalization, permanent injury, or death — must be carefully weighed against the benefits of any vaccine, even if these side effects occur rarely. No such events have been reported so far for the Pfizer/BioNTech vaccine. Such events will need careful and objective review, and a poor safety profile may prompt eliminating a vaccine from contention, even one that appears to be effective.

When, how often, how made, and how much? Safety and effectiveness being equal, the remaining questions will serve as tie breakers. How many injections are needed? How easily and quickly can the vaccine be manufactured, distributed, and stored? What is the price tag?

The cost of manufacturing will determine both the price to U.S. taxpayers, who may partially finance vaccine development and manufacturing domestically, and the feasibility of global distribution, which will remain both human rights and economic priorities until worldwide availability is established.

These trials won’t provide all the answers. A six-month study will not provide data on how long a vaccine will work. We also can’t know about very rare side effects that occur on the order of 1 in 100,000 individuals — such information becomes apparent only after vaccines are approved and in wide use. If severe, these might eventually cause a reevaluation of risks versus benefits.

Perhaps most important, the trials will not indicate how accepting Americans will be of Covid-19 vaccination programs. The most promising vaccine candidate will have a limited effect on public health if people refuse to take it. It is hard to imagine a scenario in our lifetimes for which communal participation will be more responsible for determining our collective fate.

But there is reason for optimism. There are numerous vaccine candidates, and this first favorable result augurs well for the others, especially since the early safety and immune-response data are promising.

If the timelines for Covid-19 vaccine trials are accurate, results of perhaps the most important scientific studies of our generation are on our doorstep. But the science is only the start. People respond best to scientific data they trust and understand. Let’s hope the promising results of this first Covid-19 vaccine study motivates us all to educate ourselves about how best to interpret the data. Because only through broad understanding of these results will the “warp speed” in vaccine development lead to its equally rapid acceptance and use.

Mark Siedner is a clinical epidemiologist at Massachusetts General Hospital and an associate professor of medicine at Harvard Medical School. Paul E. Sax is clinical director of the Division of Infectious Diseases at Brigham and Women’s Hospital and a professor of medicine at Harvard Medical School.