An enrapturing first date doesn’t guarantee a happy, lifelong marriage.
A single great game doesn’t instantly win a high school athlete a Division I scholarship.
One – or even two or three – hard-driving workouts won’t sculpt you into great shape.
And a few studies supporting a specific approach to treating – or avoiding – Covid-19 don’t immediately make that particular method the smart thing to do.
No, that first date leads to a second, and hopefully a third, with lots of important conversation in between. That first great game may capture the eye of recruiters and scouts, who will want to see a pattern of athletic achievement and clear signals of a strong work ethic and high character. And as for those workouts? Those first two or three are vital, not because they get the job done, but because they start a pattern that will lead to fitness – if you can sustain it.
That same concept applies to science: There is danger – both intellectually and actually – in focusing on a small group of studies that have findings you like, and then embracing them as reality.
“Science is not built on a single study,” said Sumit Chanda, an infectious diseases expert at Scripps Research and a Williamsville native. “It’s built on a body of work that is done by multiple, independent people and labs that build a consensus around our understanding.”
In this Pandemic Lessons, we explore how to know when it’s time to listen to science.
Let’s begin with some context. Why are we writing about this?
Two reasons — the first is reactive; the other, proactive:
• Confirmation bias. People have an instinctive tendency to gravitate to information they want to hear. That's a natural reaction. If you’re hesitant or resistant to getting vaccinated, any study that touts the benefits of post-infection protection – or what’s often called “natural immunity” – will sound good.
But is it good?
Maybe parts of that study are good. Maybe all of it. Maybe none.
“There may be issues, and unless you have the skill set to be able to critically review that paper, and you have expertise in that area, you may not realize that the conclusions are supported and valid or not,” said Dr. Thomas Russo, chief of infectious diseases at the University at Buffalo’s Jacobs School of Medicine and Biomedical Sciences.
Imagine walking through a densely packed housing development that has been ravaged by a storm, and your job is to identify which homes are structurally sound, which ones will need to be demolished, and which ones can be saved. You’ll need to explain your decisions, in detail, on each.
Could you do it?
If you’re an architect, structural engineer, construction professional or something of the like, you probably could. Or at the very least, you’d be equipped to explain why you can’t, because you would know what information you’re lacking by only looking at the homes from the outside.
But if you’re a writer, musician, lawyer or chemist, you probably couldn’t do this at all – short of simply taking your best guess. You wouldn’t even know what to look for, beyond the obvious (like a caved-in roof).
But of course, you’d never be asked to do this job, because construction is done by professionals.
So is science.
• Understanding the path to a better place. In a recent interview for this series, Dr. Stuart Ray, who is vice chair of medicine for data integrity and analytics at Johns Hopkins Medicine, described what it will take to proactively move our society to a healthier position as we vie to exit the pandemic.
“If transmission rates are low, then we can start taking the masks off,” Ray said. “It’s not like we’re going to say, ‘Wear masks forever.’ We just need to get to the point where people are not dying at the rates they are now, and that rates of transmission are lower.
“But right now, we’re looking at a tough winter if we don’t get higher rates of immunity. The best way to get there, scalably and verifiably, is with vaccination.”
Ray is precise in his word choice; in our interview, he even pointed out that the term “natural immunity” may misinform the public, since “the processes that generate the immune response we’re seeking” – whether initiated by infection or a vaccine – “are natural.”
“The notion that natural immunity is juxtaposed against vaccination-driven immunity might mislead people,” Ray said, suggesting we refer to protection that comes from a bout with Covid-19 as “post-infection immunity” or “post-Covid immunity.”
Clearly, Ray – like virtually all doctors and scientists – acknowledges that post-Covid immunity exists. But that’s not where this point ends, either. Being “real” doesn’t make post-infection immunity consistent, reliable, measurable or long-lasting. Scientists are still studying the durability of immunity that comes from infection, and they lack the tools to widely and adequately measure it in individuals.
That’s why Ray used another key word in his quote above: verifiably. Vaccination-induced immunity is more studied and measurable, both in the development of the vaccines and the administration of the doses. Bloomberg reports that nearly 6.7 billion shots have been given in 184 countries, and in the United States – where Pfizer, Moderna and Johnson & Johnson vaccines are approved – 407 million doses have been administered. The Food and Drug Administration is expected to soon authorize Pfizer doses for children ages 5-11, a decision that will be based on an overwhelming volume of evidence that vaccination is safe.
“We’ve already vaccinated over 100 million adults with this vaccine,” said Dr. Sharon Nachman, director of the Division of Pediatric Infectious Diseases at Stony Brook Children’s Hospital. “We’re fairly comfortable with the adverse event rate and profile, and it will be no different in children.”
It will be no different in children.
Nachman says those words based on evidence, and the expertise to understand it.
Nowadays, we can all access information on Covid-19, from vaccination data to immunity research to studies of experimental treatments like ivermectin. Google will get us the data.
But that doesn’t mean we understand it.
Why the confusion?
The volume and complexity of the research is daunting.
“There’s a lot of data, more data on one topic than I’ve ever seen in my scientific career,” said Chanda, who earned his doctorate from Stanford in 2001. “Everyone has pivoted. People who used to work on cancer or heart disease have pivoted to Covid. There’s a lot of data out there, but a lot of the data is conflicting.”
This is normal. Much like lawyers may take distinctly different approaches to strategizing a case, or architects will create unique designs for a proposal with exacting cost and size specifications, the way one scientist designs a study will differ from the next. For example, a Cleveland Clinic study of post-Covid immunity focused on health care workers. The results support the strength of this natural form of protection, but Cleveland Clinic researchers have also pointed out that the people studied were “younger and healthier than the general population.” The researchers also continued to advocate getting vaccinated.
A much-publicized Israeli research paper also supported post-Covid immunity, while a widely reported Kentucky study emphasized the greater protection afforded by vaccination.
Sorting, debating and reconciling that conflicting data is the work that scientists do. This isn’t new, but now it’s happening in the public eye. Scientists have long posted draft copies of their research – called “preprints” – on publicly available websites. This allows their findings to be reviewed by their fellow researchers, who will give feedback and may request additional research or data.
This peer-review process “is the foundation that science stands on,” Chanda said, and it’s a prerequisite for getting published in prestigious journals, some of which reject 80% to 90% or more of the papers submitted.
“That’s because what the authors are claiming just doesn’t stand up to scrutiny when you have independent people looking at it,” Chanda said. “That’s exactly the kind of checks and balances system that keeps science rigorous and productive.”
This sounds like a good thing. Isn't it?
Yes. It just demands perspective.
“Covid changed everything, because now there’s a sense of urgency,” Russo said, noting that the scientific review process “can take a while,” and quick sharing of information during the pandemic allows researchers to share data to both question and build on each other’s work.
“The problem is if you then take the conclusions of all of those preprint papers and assume they’re correct,” he said. "They’re not always going to be correct, so that may lead you down a path that is not valid, and it’s not scientifically justified.”
This happened over the summer with a study from Egypt lauding the use of ivermectin, an antiparasitic drug available by prescription to humans, and used as a deworming medication for animals. The Egyptian researchers found it reduced Covid-19 deaths – a seeming revelation that garnered worldwide attention. But then the paper was flagged by other scientists as having serious flaws in the research design and for data manipulation and plagiarism. The website that published the paper ultimately rescinded it.
That’s an example of the danger in embracing a single study.
What, then, do scientists look for?
“What the bulk of the literature shows,” said Russo, offering this example: You have 10 papers that study how to manage a specific medical condition. Eight support one method. One suggests that approach is incorrect. Another is neutral.
If all of the papers are “of equal quality” and study “a comparable patient population,” Russo said, “the bulk of the literature supports that you should go ahead with that treatment.”
Likewise, he added, if you have several studies on a topic, but only two are high-quality, controlled and well-designed studies, they will outweigh several more “small, poor quality trials as well.”
Sound complicated? It is. That’s why the best move for most of us isn’t to play scientist. Rather, it’s to listen to them – a lot of them.