Maintenance Phase Reality Check: "COVID Conspiracies"

Initially, when this episode was released, I didn’t plan to do a fact check of it. I was told that Aubrey and Michael had an expert fact-checker, which thrilled me, and I figured it would be pretty truthful. Unfortunately, that’s not the case. 

This episode is bizarre for a couple of reasons. First, while Michael and Aubrey attempt to discredit scientific institutions, they actually end up revealing more about the weaknesses and deficits in their own industry (journalism), particularly as it pertains to the spread of misinformation (which they are complicit in). They repeatedly provide examples of individuals outside the scientific community (e.g., Joseph Mercola, James Todaro) as evidence of the scientific establishment failing the public. It is absolutely correct to cite these folks as sources of misinformation, but to incorrectly link them to the scientific community only serves to discredit actual science. The real culprit with respect to these quacks was the media, not the scientific institutions (which were never paying the quacks any attention). 

Additionally, in their effort to explain how misinformation spread during COVID, Aubrey and Michael engage in the exact same activities that caused that misinformation to be spread. For example, they cite a preprint as “a very good scientific article” without any mention of it being a preprint. It is well-established that preprints were a substantial source of the rampant misinformation during the early days of COVID. Another example, not unique to this specific episode, is that Michael curates quotes such that they conveniently leave out information that run counter to his intended narrative. Additionally, sometimes Michael and Aubrey read quotes verbatim from articles that they don’t cite or credit in any manner. 

While anyone would be hard pressed to argue that US government health agencies handled the pandemic well, there are some things that Michael and Aubrey try to dunk on in this episode that just aren’t dunkworthy. There are very real criticisms to delve into with respect to the FDA and the scientific publishing world, but Aubrey and Michael don’t have the requisite knowledge to do this, and instead they perpetuate some incorrect and problematic beliefs about a system they fundamentally don’t understand. In the meantime, they reveal the very real schism between journalists and scientific communication experts. Not just any journalist can do scientific communications well, and Michael and Aubrey are simply not trained enough or diligent enough in their research for this podcast to be trustworthy sources of scientific information. (If you’d like to listen to an example of journalists covering a scientific dispute with nuance and humor in the way that I wish MP would, I highly recommend the Nerd Fight episode of Endless Thread.) Of course, this episode also suffers from the common MP issues of misrepresentation of scientific studies and reporting incorrect numbers. It also veers into dangerous territory, as Michael says that veterinary ivermectin has a similar formulation to human ivermectin and effectively disregards the risk of taking it, as long as it is in appropriate doses. I hope no one needs to hear this, but please don’t take veterinary ivermectin. It’s not safe, no matter what Michael says. 

---

Ok, on to the episode:

Michael: This paper is retracted within two weeks, but this then results in a wave of articles, one of which, this is on Joseph Mercola's website, he says, “Is SARS-CoV-2, a chimera virus built from HIV, flu, and SARS”?

As noted in my intro, Joseph Mercola is a known quack. No one from the scientific community would consider him an expert or reliable source of information about anything.  

The day after the emergency declaration, we get an article in the Lancet by a bunch of doctors who basically were looking at the data coming out of the hospitals in China, and they notice that of the patients who were hospitalized with COVID, 30% of them had hypertension and 12% had diabetes. And that's slightly higher than it is in the population like higher than you would expect. And so, they write this article saying, “We know that when people have these conditions,” one of the things they typically take is ibuprofen. 

This is false. The article is actually a “Correspondence”. This is an important distinction because in the Lancet, a “Correspondence” is not peer reviewed. The Lance describes a Correspondence as, “Our readers’ reflections on content published in the Lancet journals or on other topics of general interest to our readers.” The fact that Michael leaves that out when describing this article is problematic. 

Anyway, this Correspondence reports that diabetes and hypertension were the common comorbidities among patients with poor outcomes in three different studies. Specifically, in the first study, 22% of patients who died in the ICU from COVID had diabetes and 22% had cerebrovascular disease. In the second study, 16.2% of patients with severe disease had diabetes and 23.7% had hypertension. And in the third study, 12% of patients admitted to the hospital with COVID had diabetes and 30% had hypertension. It’s unclear why Michael cherry-picked just the results from the third study, given that these are all presented in the same paragraph. The authors then wrote, “Notably, the most frequent comorbidities reported in these three studies of patients with COVID-19 are often treated with angiotensin-converting enzyme (ACE) inhibitors; however, treatment was not assessed in either study.” 

This is a reasonable enough thing to wonder, especially in a Correspondence which is purely an opinion piece. ACE inhibitors could be a confounder that accounts for why people with hypertension and diabetes have poor outcomes in these studies. The authors also provide some biological rationale for this. Again, this is a totally reasonable scientific process for developing a hypothesis. 

And there is some evidence that for other respiratory illnesses, taking anti-inflammatories can actually reduce the activity of your immune system. And so, it's worth looking into, were these patients taking anti-inflammatories? It says in the article, “If this hypothesis were to be confirmed, it could lead to a conflict regarding treatment.” They’re just purely speculative. 

This is not what the Correspondence says. The authors lay out the scientific rationale for this hypothesis which is based on 

1) the fact that there is upregulation of ACE2 in both hypertension and diabetes and, 

2) the fact that coronaviruses bind to target cells via ACE2. Again, very reasonable. 

There is nothing here about anti-inflammatories and immune system activity. This is not to say that there weren’t other scientists or articles about that. But it’s false to lump that into this narrative. And again, yes, it’s speculative because this is a Correspondence. 

These are not people in China. These are not people who are working with COVID patients. They're just like, “Hey, people should know, like, this might be at play.” But then, of course, this gets taken up by random people and starts bouncing around online as like, don't take ibuprofen. So, a version of this ends up getting tweeted out by a French doctor who's like, an Instagram influencer, something, something. He's like, “We know ibuprofen is associated with worse COVID outcomes,” which is not true. 

Michael’s reaction here reveals his lack of understanding of science. This hypothesis and opinion that was put forth by these doctors does not rely on being on the ground in China, treating patients. That’s irrelevant.

Michael: Exactly. But then I think one thing that's interesting about this is the relationship between the institutions of public health and these conspiracy theories that run around. I don't know if I would even call this a conspiracy theory. It's more like just false information that goes around. But it's like, I don't think that the actual institutions of public health were as prepared for this as they should have been.

It seems wild to blame public health institutions for bad journalism, but I understand that as journalists, Michael and Aubrey are biased here. 

Michael: This gets debunked. A couple days later, the WHO puts out better guidance. They're like, “Actually, we don't really know this is like super hypothetical. It might turn out to be true later, but right now we can't really say anything.” It added to this sense at the time that it was just there's so much stuff going around. Like, everybody, including me, probably should have been more careful about being like, “Make sure you don't take ibuprofen.” I was like, texting friends. I was like, “If you're taking ibuprofen, don't take it.”

Nothing to debunk here. It was an unproven hypothesis that was presented as such (except by a rogue French Minister of Health). It’s worth noting that the FDA immediately issued a guidance following the Lancet Correspondence that said, “At this time, FDA is not aware of scientific evidence connecting the use of NSAIDs, like ibuprofen, with worsening COVID-19 symptoms.” But it doesn’t work with Michael’s narrative very well to include examples of the public agencies acting appropriately. 

Michael: For this very preliminary study, we get a press release titled possible coronavirus drug identified. Ivermectin stops SARS-CoV-2 virus growing in cell culture. And then the first paragraph of the press release is, “A new study has shown that an antiparasitic drug already available around the world can kill the virus within 48 hours. Scientists found that a single dose of the drug Ivermectin could stop SARS-CoV-2 growing in cell culture. The next steps are to determine the correct human dosage, ensuring the doses shown to effectively treat the virus in vitro are safe for humans.” So, this does say very clearly, like, this isn't a cell culture, but also it sounds pretty promising.

Michael is misrepresenting the press release for his own agenda. What he just read is the “summary”, not the press release itself. The ACTUAL press release says (emphasis mine):

A collaborative study led by the Monash Biomedicine Discovery Institute (BDI) with the Peter Doherty Institute of Infection and Immunity (Doherty Institute), a joint venture of the University of Melbourne and Royal Melbourne Hospital, has shown that an anti-parasitic drug already available around the world kills the virus within 48 hours.

The Monash Biomedicine Discovery Institute's Dr Kylie Wagstaff, who led the study, said the scientists showed that the drug, Ivermectin, stopped the SARS-CoV-2 virus growing in cell culture within 48 hours.

"We found that even a single dose could essentially remove all viral RNA by 48 hours and that even at 24 hours there was a really significant reduction in it," Dr Wagstaff said.

Ivermectin is an FDA-approved anti-parasitic drug that has also been shown to be effective in vitro against a broad range of viruses including HIV, Dengue, Influenza and Zika virus.

Dr Wagstaff cautioned that the tests conducted in the study were in vitro and that trials needed to be carried out in people.

"Ivermectin is very widely used and seen as a safe drug. We need to figure out now whether the dosage you can use it at in humans will be effective -- that's the next step," Dr Wagstaff said.

Though Michael presented this as a somewhat reckless scientific conclusion, in reality, the scientist was being quite careful in his speculation about ivermectin in humans.

Michael: But the problem with this study is basically the amount, to get the amount of Ivermectin that would be equivalent to the amount that they used in this Petri dish. I've seen different numbers. One of them says you would have to ingest around two and a half pounds of Ivermectin.

This sounds a lot like the conspiracies that were going around during COVID. See the above full text of the press release in which the scientist makes it abundantly clear that the dosing in humans would be different.

Michael: So, there's a very good scientific article with just a timeline of the rise and fall of Ivermectin. So, this is from that.

The article that Michael is citing here is a preprint. That Michael calls this a “very good scientific article” without acknowledging that it is not peer-reviewed shows his lack of understanding or care for scientific rigor. This is especially important given the topic he is covering in this episode, because preprints were a substantial source of misinformation in the early days of COVID. For those who aren’t aware, a preprint is a scientific paper that is freely available online prior to any peer-review process. One article sums this up nicely: “With maximum media coverage, and in the absence of effective treatment, journalists were on the lookout for the slightest piece of information, verified or not. The information contained in preprints has been spread to the general public by both official and unofficial media, without reference to their distinction from peer-reviewed papers.” Michael is contributing to the problem by doing the same thing here.

Aubrey: I mean, I think we've talked about this a fair amount with weight loss studies. That there's got to be a level of acknowledgment from researchers around weight loss that their research will get introduced into a context where people are going to figure out how to monetize it as quickly as possible, and people are going to take it as much more declarative than it necessarily is.

This is such a bizarre statement coming from a podcast that cherry picks data and chooses to highlight the media about an article instead of what the authors say, themselves. Aubrey and Michael are the voices that are amplifying the problematic messaging, rather than the ones actually paying attention to what the scientists say. Point in case, see the passage above in which Michael misquotes a press release to omit the caution from the scientist who led the study.

So, in the months after the Wall Street Journal editorial, we get a randomized control trial out of Egypt that shows a 90% reduction in death rates from Ivermectin. We also get a study out of Brazil showing a 70% to 85% reduction in deaths. And these are both a huge deal at the time. These are effects roughly on par with the vaccine. But eventually, people circle back and find out that both of these studies essentially could not have happened.

A few wrong things here. First, the Egyptian study was a preprint. The Brazilian study reported a 92% risk of death. It was published in a journal that is very controversial in the scientific community because it has “expedited” peer review. In other words, these are examples of the media and the public going crazy without scientific consensus or support. Additionally, Michael’s statistics here about the vaccine are wrong (again, not sure where he got these numbers from). First of all, the COVID vaccine trials were looking at prevention of symptomatic disease, not death from COVID. I’m going to guess that Michael looked at the Pfizer vaccine clinical trial results which found that the vaccine efficacy rate was 95%. However – and this reveals his lack of scientific literacy – the outcome in these trials was symptomatic disease. Obviously, ivermectin does not work to treat COVID. But it’s also wrong to say that the COVID vaccines reduce deaths by 95%. It’s important to get the facts right.

Michael: There's a good article in the Atlantic by James Heathers, who's one of the people who does this forensic analysis of statistics. And he says, “You know part of it is understandable early in the pandemic, it was just like, let's fucking throw everything that could work at the wall. Some bad studies are going to get through in a context like that. This isn't really anybody's fault. But it's like the entire process of peer review is, of course, unpaid. And reviewers have to take the data at face value. It's like, according to the data that you have in your paper, is this methodology sound? Is your analysis sound? Did you do the statistics right, etc. They don't have the resources to check is your data fucking fake?

While this is true, the larger issue here was preprints, as I discussed above. And most of the papers Michael is discussing were preprints. As someone who is well-versed in the peer review process – which Michael is not – I can tell you also that it is ultimately up to the editor, not the reviewers, whether a paper gets through. The issue is not about being paid – it is that the reviewers are not necessarily experts in the methods or disease area of the study they are reviewing. Additionally, most data are proprietary and not available for reviewers to examine (if they even had time to do that).

Aubrey: Not a failing of individual responsibility of people who are reviewing these studies, not a whatever. But just like, we actually don't have a system to handle this thing. 

Michael: Right, right. I mean, that's the thing. It's like so much of this is just a basic resources gap.

Michael: As we move forward into a world where everyone can say anything at any time on the Internet for free. All of these institutions really have to double down on, like, if you read this in the Lancet or wherever, it is true, it is fact checked and be transparent about the processes that go into these things. Dedicate huge resources to, yeah double and triple checking these things. Especially, if we’re talking about something like Ivermectin, where it’s like we’re in the middle of a once in a century pandemic, a study that indicates, “Hey, this is a cure.” Of course, people are going to take that up and start taking fucking Ivermectin.

This whole part of the conversation shows how little Michael and Aubrey understand about the scientific publishing process (it’s also just such a bizarre take for a podcast that consistently gets basic facts wrong and has no fact-checking mechanism). This article contains a great infographic for those who want a very simple overview. We do indeed “have a system to handle this thing” - it’s called peer review. But it’s very broken. This is not a new take - it’s the proverbial dead horse in the scientific community. There have been countless articles/editorials/Tweets/blog posts detailing the flaws of the scientific publishing process going back decades (ref 1, ref 2, ref 3, ref 4, ref 5, and there’s more where those came from). And yet, we’ve seen minimal changes in the system, and none at scale. Why? Power, money, politics. And the fact that very few of the people who have written these pieces put forth a solution (although this is starting to change - there are some interesting ideas being discussed right now that are promising!). As we all know, it’s very easy to sit back and criticize something, but coming up and implementing a better solution is hard, especially when the system is deeply entrenched and there are powerful forces with a lot of capital that don’t want the status quo to change (i.e., publishers).As Michael Eisen wrote in the Guardian in 2003, “Scientific publishing is incredibly lucrative. How could it not be? Publishers are given (for free) incredibly valuable content by research scientists who are then compelled to pay whatever the journals ask to access this content. Elsevier - the largest publisher of scientific journals - has profit margins of more than 30%. Publishers will fight hard to hang on to this business.” Anyway, TLDR: journals have plenty of resources - they just have no motivation to change. 

Relevant suggested listening: Iron Culture: Ep. 270 - Scientific Retractions Aren’t Working (libsyn.com)

Michael: -paste slurping. [Aubrey laughs] So, based on that and the right wing basically taking this up, it's like, well, we know it cures COVID. We have this huge spike in prescriptions. So, before the pandemic, there were around 4000 prescriptions of Ivermectin per week. At the peak of this myth, in August of 2021, there were 40,000 prescriptions per week. So, a tenfold increase.

False. At the peak in August 2021, there were 88,000 prescriptions per week.

And because animals cannot swallow giant ass pills, this is usually given as, like, a paste or what is called a drench, which is like, they stick a tube down the cow's stomach and pump it with Ivermectin. And so, what starts happening when people can't get prescriptions for Ivermectin from their doctor is they then go to pet stores or animal feed stores and they get animal Ivermectin, which appears to be roughly the same formulation.

False. It is a different formulation. I’m actually pretty shocked that Michael would even say this – it’s very dangerous to suggest they are the same. 

Michael: There were, at the time, two deaths in New Mexico of people that just took way too much Ivermectin, mostly because it was like this livestock dosage and their kidneys failed and they died. So that's actually true. These poisonings were happening. But it's also very important to point out that most of the poisonings were relatively minor. People had gastrointestinal stuff or they felt shitty for a couple of days. Ultimately, they were fine. The normal dose of Ivermectin, like, what you would take if you needed it for anti-parasite anti-scabies, is totally safe. People were going to livestock stores and getting animal doses of Ivermectin. That was happening. But on nowhere near the scale that it seemed like if you were around social media at the time.

At one point, the FDA puts out a tweet from its official account that says, “You are not a horse. You are not a cow. Seriously, y'all, stop it.” I really object to the FDA doing this, both entrenching the idea that this was happening on a much larger scale than it was and for mocking the people who were doing that.

Michael is minimizing the issue – just because the one Rolling Stone article he looked at was not fact-based doesn’t mean that there weren’t real problems with ivermectin. And it’s not the FDA’s responsibility to only call out things that are causing people to die - it’s their job to tell people not to ingest things that are dangerous for them. If they hadn’t called this out, Michael would have said they weren’t acting appropriately. It is wild that Michael is saying the FDA should not have told people not to take veterinary ivermectin. Many of the patients who had ivermectin toxicity had to be treated in an ER or go to the hospital (ref 1, ref 2). This take is so strange in light of MP’s take on semaglutide and the way they treat the side effects of Wegovy/Ozempic which are generally quite mild, as well. For those drugs, MP uses the side effects as proof that they shouldn’t be used, while here Michael is like, “eh, it was mostly mild.” I also think it’s important to point out that Michael himself was a promoter of what he now is saying was fake news. See below:

Michael also previously had very different beliefs about the conspiracy theories. Rather than saying these people were victims, he is quoted in an NY Magazine article saying, “The media treated the lab leak as a conspiracy early in the pandemic because the people pushing it were conspiracy theorists. Zero Hedge, Trump, deranged message boards.”

It’s also important to point out that Michael’s entire media persona is mocking people. To call the FDA out for mocking when your podcast’s Twitter page posted a meme mocking people who believe drinking red wine is healthy because of the antioxidants and reposted a Tweet about a US Representative’s wife DYING, is hypocrisy at its finest.

So, we're rewinding to the beginning of the pandemic again when we have a tweet from James Todaro, MD that says, “There's growing evidence of chloroquine as a highly effective treatment for COVID-19 in a collaborative effort, Gregory Rigano, John Hopkins, Thomas Broker, PhD, Stanford and I explore chloroquine as a treatment/prophylactic to treat and prevent coronavirus.” And with this tweet, he also includes a link to a Google Doc which I am about to send you. 

Ok so, again, this guy is a conspiracy theorist - not someone who scientists would take seriously. He believes the 2020 election was rigged James Todaro, MD on X: "The 2020 US Presidential election far exceeded 2 standard deviations of historical voter turnout over the past ~100 years. A statistically improbable outcome. For comparison, both the 2012 and 2016 elections with Obama were well within a single standard deviation. https://t.co/mcivSZ4qYW" / X (twitter.com) No scientists ever took this guy seriously. 

Michael: I've read large chunks of this. It's like they're really leaning into jargon. A little bit further down It says, “The cell surface expression of under glycosylated ACE2 and its poor affinity to SARS-CoV spike protein may be the primary mechanism by which infection is presented by drug pretreatment of cells prior to infection.”

Ok so obviously this paper is BS, but Michael, this is just a scientific paper. It’s not “leaning into jargon”, it’s using scientific language appropriately. The really interesting story here is just how bogus all of this is. Here’s a great Wired article about it: Chloroquine May Fight Covid-19—and Silicon Valley’s Into It | WIRED. 

It's also true that it doesn't appear he was ever a practicing doctor.

He is a practicing ophthalmologist. You can easily look this up: Dr. James M. Todaro, MD | Dearborn, MI | Ophthalmologist | US News Doctors. It’s even on his LinkedIn

I am going to send you a description.

Michael and Aubrey continue to read verbatim from this article, which they don’t cite nor do they put in the list of links for the episode: The Strange Origins Of Trump’s Hydroxychloroquine Obsession | HuffPost UK News (huffingtonpost.co.uk)

Michael: I will say one of the key differences between Ivermectin and hydroxychloroquine is that, as we said, Ivermectin is relatively safe. If you take it at the doses that most people take it at, it's basically fine. There aren't a lot of side effects. Hydroxychloroquine has very well-documented side effects. So, if you have a heart condition, taking hydroxychloroquine can be dangerous because it can cause extra arrhythmias. And so, if you have an existing heart condition, you shouldn't be taking this. This is very well known, very well documented. So, it's not simply the case that let's just give this to everybody in America, and then we all won't get COVID.

For everyone who says that MP is not a science show and isn’t doing any harm with the misinformation or “mistakes” they make, I’d like to highlight that Michael is absolutely out of his bounds here, speaking as a medical expert. While there are only mild side effects associated with ivermectin when it is taken by individuals who were prescribed it appropriately, that does not at all make it safe for the general public. There is also a risk of some pretty serious neurotoxicity. And taking a formulation for animals is a whole other issue. It’s so irresponsible for Michael to minimize the risk of ivermectin like this. Really disturbing.

Michael: This results in an avalanche of people pulling back recommendations. So, earlier, the FDA had actually issued an emergency use authorization for hydroxychloroquine. Like, yeah, why not. Start giving it to people, whatever. It seems to be relatively widely prescribed. In June, just after this Lancet report is published. There's also a New England Journal of Medicine report. After these reports come out, the FDA pulls back that advice and is like, “We really shouldn't be giving this to patients until we set it more.”

First of all, the FDA never advised using hydroxychloroquine in this population. This is a good opportunity to explain what an Emergency Use Authorization (EUA) is and is not, as Michael doesn’t seem to understand. Per the FDA (emphasis mine): “In certain types of emergencies, the HHS Secretary may issue a determination and declaration under the Food Drug and Cosmetic Act that permits FDA to issue emergency use authorizations (EUAs) to facilitate access to medical countermeasures (drugs, biologics, vaccines, and devices) that can be used to diagnose, treat or prevent a serious disease or condition in a public health emergency. Products authorized for use in this way may not be approved by FDA for any use, or they may be approved for other uses but not for the emergency use. FDA decides whether the use of the product is likely to more helpful than harmful for the emergency use; i.e., the agency determines that the known and potential benefits of the medical products for their intended uses outweigh their known and potential risks. This authorization is reserved for emergency situations and is NOT the same as FDA approval or licensure.” The EUA stated: “Because chloroquine phosphate and hydroxychloroquine may possibly help very sick patients, FDA is allowing these drugs to be provided to certain hospitalized patients under an EUA issued March 28, 2020. Under the EUA, health care providers and patients are provided with information about the risks of these drugs. However, more data from clinical trials are necessary for us to determine whether chloroquine phosphate or hydroxychloroquine sulfate are safe and effective in treating or preventing COVID-19.” While Michael would like you to believe that the FDA is flippant and corrupt, this was following the best scientific evidence available at the time. The way the FDA handled this was actually impressive, given the overall poor handling of COVID by the US government. 

The WHO cancels its own trial. So, basically, the entire medical establishment goes from, this is a promising treatment to this is dangerous and you shouldn't be giving it to patients almost overnight. Mostly on the basis of this Lancet study.

Completely false and totally oversimplified. The WHO paused recruitment and then subsequently resumed the trial. The hydroxychloroquine arm was later closed due to interim report findings of lack of effectiveness. It was not canceled because of this Lancet paper. The same is true of the Recovery Trial which was published in NEJM.

So, this May 2020 Lancet report, if we look back, this Lancet report surveys 96,000 patients in 671 hospitals and finds that the death rate is much higher. After this comes out, a huge number of researchers are like, wait a minute, how did they get data on 96,000 patients in May of 2020? So, the way that the study worked is there is a company called Surgisphere, which is harder to pronounce than hydroxychloroquine. [Aubrey laughs] This Surgisphere company gathered up data from all of these hospitals, basically created this massive database. And then researchers can dive into it, slice and dice the data however they want. And so, the head of this Surgisphere company is listed as a co-author on a lot of the studies that start coming out using this massive trove of data. 

It's not clear to me if it's an academic or a journalist, contacts one of the largest hospitals in New York City, where a lot of the early patients would be. They are like, “Okay, you know, how did you give your data, what format did you give your data over to Surgisphere?” And the hospital's like, “Who's Surgisphere?” Somebody asks Surgisphere, like, “Hey, sorry, do you mind giving us just a list of the hospitals? Like, we're trying to kind of double check this.” And Surgisphere is like, “No,” we won't tell you which hospitals give us our data.

I have no idea where Michael got this narrative of events, but it’s not what any of the articles about this say (including, importantly, the ones they linked to). He is off-base here in terms of what the issues with this study were. First, having 96,000 patients is not particularly impressive. There were already 1 million confirmed COVID cases as of April 4, 2020. The actual issues with this paper were much more problematic. Mondher Toumi (a very well-respected academic) wrote a really great paper outlining the issues with the Lancet paper: A critical analysis and review of Lancet COVID-19 hydroxychloroquine study - PMC (nih.gov). TLDR: there were multiple methodological flaws with respect to treatment of missing data, the model was incorrectly specified, outcomes were reported that would not have been available in electronic medical records, the data were full of anomalies that suggested either selection bias or compromised data quality, and results were inconsistent with established scientific knowledge.

Michael: What is amazing to me about this is, this is a scandal in which the scientific establishment worked.

Aubrey: I do think we have to stop viewing retractions as failures and more as an example of the system working well.

I don’t think a major journal publishing a paper based on fraudulent data is the “scientific establishment” working. And Aubrey’s comment here just reveals how little they understand about academia and the scientific publishing process. If things worked, papers like this wouldn’t make it into journals. Journals would not profit off the unpaid labor of “peer” reviewers. And politics would not play into what research gets funded and what doesn’t. The bottom line here is that Michael and Aubrey don’t have the experience or insight required to comment on this. 

Michael: Some of this is like dense ideological rich text. And some of this is just like America's poor public education system that people do not understand how viruses work. You can make little connections, correlations, whatever wellness, immune system fine. But it's like taking zinc will not prevent a virus from-- it's little like spike proteins, like linking into your cells. That just isn't the way that it works guys.

I’d argue a lot of this is also because people get their health and wellness advice from podcasts like Maintenance Phase which purport to know things that they don’t. What’s especially frustrating here is that Michael is suggesting that people don’t understand how viruses work while simultaneously making it very clear that he does not understand how viruses or the immune system work yet without admitting it. First of all, it’s worth noting that spike proteins are not present on all viruses - they are distinct to coronaviruses (that is where the name coronavirus comes from). Second, there were a few hypothesized mechanisms through which zinc might reduce COVID severity, and preventing “like linking into your cells” was not one of them. If you’re going to be pedantic, at least do your research first. 

Michael: There's actually like a long history of bizarre hype around vitamin D, especially like taking vitamin D supplements. So, there's a weird wave of vitamin D hype in the 2010s, I'm sending you a fucking cursed paragraph about this. This is from a New York Times article talking about the hype.

Aubrey: Dr. Mehmet Oz has described vitamin D as, “The number one thing you need more of,” telling his audience that it can help them avoid heart disease, depression, weight gain, memory loss, and cancer. And Oprah Winfrey's website tells readers that, “Knowing your vitamin D levels might save your life.” Mainstream doctors have also urged Americans to get more of the hormone including Dr. Walter Willett, a widely respected professor at Harvard Medical School.

Michael: Yeah. So, we have been having these overblown claims about vitamin D for like, a very long time. There's a fascinating 2018 New York Times article about this one guy who wrote a book called The Vitamin D Solution. So, I'm going to send you the first couple paragraphs of that.

Aubrey: Dr. Michael Holick's enthusiasm for vitamin D can be fairly described as extreme. The Boston University endocrinologist elevates his own levels of the stuff with supplements and fortified milk. When he bikes outdoors, he won't put sunscreen on his limbs.

Again, citing Dr. Oz as evidence that doctors/scientists think anything is just silly. Also, this quote has been altered, again. The full second sentence says, “The Boston University endocrinologist, who perhaps more than anyone else is responsible for creating a billion-dollar vitamin D sales and testing juggernaut, elevates his own levels of the stuff with supplements and fortified milk.” I really don’t understand why Michael doesn’t acknowledge when he paraphrases or alters quotes.

Michael: It's also, it's so funny to me that he's like, dinosaur like, what if their bones were weakened due to lack of vitamin D? It's like the bones are the only thing we have from the dinosaurs. [Aubrey laughs] That's literally all the evidence of dinosaurs that we have is the bones. So, we can very readily check the bones.

Oh Michael, this is almost as absurd as Holick’s statement. As the Smithsonian says, “Everyone learns in school that that dinosaur skeletons are bones turned to stone.” But, to use a page out of Michael’s playbook, maybe Michael’s school didn’t properly educate him about fossils. Anyway, we certainly have other things, like soft tissue, in fossils. But mostly we don’t have actual bones. 

The vitamin D truth or narrative really ramps up in April of 2020 with an article called “Patterns of COVID-19 Mortality and Vitamin D, an Indonesian study.”

Again, this was a preprint.

So, this is a study where they take 780 patients at an Indonesian hospital. And just like they do with these other studies, they look through a huge database of everybody's characteristics of age, preexisting conditions, time of entry, time of discharge, etc. And they look at, like, people who died and people who didn't die, and they're like, okay, what are the differences between these two people?

This is an oddly non-scientific way to describe a totally normal method for a study. It’s a fraudulent paper, sure, but extracting demographic and clinical characteristics from electronic health records for patients with a confirmed diagnosis is commonplace. 

Michael: And so, when they look at the data directly, they find that people who have high vitamin D levels, people who have enough vitamin D, are a lot less likely to die. And then they start controlling for things like age, preexisting conditions, and the association becomes even stronger.

Nope. This would have been a good place for Michael to get an expert opinion. Here’s what the authors actually did:

First, they did what we call univariate tests. This means that they looked one at a time at each of the variables (e.g., age at diagnosis, sex, etc.) that they thought might be related to the outcome (death, in this case). This is what we call an “unadjusted” or “crude” analysis. We’re looking at the relationship between two things (e.g., age at diagnosis and death) without accounting for any other potentially relevant variables. This is a pretty common thing to do, though some epidemiologists and biostatisticians (including me) would say it’s not useful in the instance, so I’m not entirely sure why the authors included this. But anyway, this is normal. 

Ok, next, the authors conducted another prespecified analysis that looked at how vitamin D level category (deficient, insufficient, and sufficient) was associated with mortality. But for this analysis, they wanted to account for the other variables that are relevant (in this case, they only had data on age at diagnosis, presence of comorbidities, and sex). This is what we call an adjusted analysis, and provides (ideally) an estimate of the association (statistical relationship) between the exposure of interest (vitamin D level category) and the outcome of interest (mortality) while taking into account other things that might affect that association. Contrary to what Michael says, the authors didn’t just “start controlling for things.” This was a prespecified analysis and it is entirely standard and appropriate. This is how you get away from the phrase that Michael and Aubrey love so much: “correlation =/= causation.” I have a separate short post about how Aubrey and Michael (and others!) misuse this phrase and explains more about what “statistical relationship” means.

Michael is also misreporting the study. The adjusted analysis actually reduced the magnitude of the association between vitamin D level category and mortality (this is to be expected). In the unadjusted analysis, the odds ratio for mortality comparing vitamin D insufficient patients to vitamin D sufficient patients was 12.55, while the adjusted odds ratio was 7.63. Additionally, in the unadjusted analysis, the odds ratio for mortality comparing vitamin D deficient patients to vitamin D sufficient patients was 19.12, while the adjusted odds ratio was 10.12. 

It's cited in the British Medical Journal. It is cited by-- There's this body as part of the NHS in the UK that does evidence reviews of, what drugs are we going to cover? We're going to look at all the evidence. It's called NICE. It's cited in a NICE report on treatments for COVID-19…

Nope. Michael is probably going off this Letter to the Editor (because he reads from it verbatim later without citing it) that suggests that an Editorial piece in BMJ cited the article, but you can see the list of references and it’s not there. It also was NOT cited by NICE, as that Letter to the Editor and Michael posit. 

Vitamin D is not routinely checked in Indonesia. Data collection method was retrospective, which is suspicious.

Just to clarify, because it’s not clear that Michael understands this, there is nothing suspicious about retrospective data, in general. Many many many studies are conducted using retrospective data. What is suspicious here is that vitamin D is not routinely checked in Indonesia, so if the authors wanted to study this, they would have been unlikely to have found the vitamin D data in the electronic medical records - it would have required designing a study to collect those data prospectively. 

Michael: The thing is, I actually find this very chilling. This is being cited in the British Medical Journal and by the NHS and affecting policy. And no one fucking googles the authors.

Again, neither of these things is true. Michael didn’t check the facts before he made this podcast. 

Michael: And people point out, even at the time, it's like, “Oh, it's not peer reviewed, etc., but it's like, in a fast moving, deadly pandemic, you need to get information out as quickly as possible. A lot of preprints during COVID turned out to be true and turned out to be really important. And so, you don't want to have something where every single thing during a deadly pandemic must be triple checked. That's not workable. But also, the problem with especially these conspiracy narratives is that there's, like, there's so many bad faith actors out there.

I’m not entirely sure what Michael means when he says that a lot of preprints “turned out to be true” or “really important,” or how he is coming to that conclusion. As I’ve already noted, Michael does not have enough expertise on this topic to weigh in. Most estimates suggest that only about 50% of COVID preprints were later published at all, and many of the studies that were later published had some changes from the preprint version. Of course, the primary issue with preprints, as this episode has perhaps unintentionally highlighted, is how poorly they are reported on by the media. A great article that reviewed media stories about COVID preprints found that only 37.6% of media studies that discussed a preprint even mentioned that the study was not peer-reviewed and only 21.7% mentioned that it was a preprint. And Michael is perpetuating that in this episode, by failing to mention when the papers he refers to are preprints. I agree strongly with Ravinetto et al. who wrote: “Therefore, we contend that all preprint servers should state, as does MedRxiv, that “Preprints are preliminary reports of work that have not been certified by peer review. They should not be relied on to guide clinical practice or health-related behaviour and should not be reported in news media as established information.” Michael should take this to heart.

Michael: So anyway, this comes out in April of 2020. It's not debunked until a couple of months later. So, there's this what appears to be this trickle of information coming out that vitamin D has some connection with preventing COVID, preventing deaths from COVID. There's also a really interesting scientific debate about country correlations. So, people start looking at COVID case rates, COVID death rates, etc., and then correlating that with the vitamin D levels in the population. And they find there's the first paper is like, well, countries with higher vitamin D levels don't have as bad COVID outcomes. But then another paper comes out that controls for different things and they find no significant results. 

The article wasn’t really “debunked” in that it was never considered to be “proof” of anything within the scientific community. Michael and Aubrey don’t seem to understand this, because they aren’t scientists, but the scientific community looks for where the majority of the evidence points, not just a single paper. I have also not seen any speculation that vitamin D prevented COVID - though there certainly was speculation that it could treat it or improve outcomes. It’s really interesting that Michael is reporting this as if it was a widespread scientific phenomenon, because it was really just a small group of researchers. 

Also, to be abundantly clear, because Michael continues to omit this very important detail, the country level vitamin D levels and COVID outcomes paper he is referring to here is a preprint. It is chock full of issues (hence why it was never published). It would be helpful if Michael would include these references in their links. I don’t know why they continue to cite papers and not include them for their listeners. 

Side note: I am refraining from going into detail about how Michael’s use of the term correlations here belies his lack of knowledge of study design and statistics, but I will provide some explanation at the end, for those who are interested. In short, correlation =/= “analysis”. It’s a very specific type of analysis. 

Michael: Exactly. Yeah. So again, at this time in summer of 2020, there's some reason to believe vitamin D could be helpful for COVID. So, on August 29th, 2020, we get the first randomized control trial that tests vitamin D and COVID outcomes. It is called Jesus fucking Christ “Effect of calcifediol treatment and best available therapy versus best available therapy on intensive care unit admission and mortality among patients hospitalized for COVID-19: A pilot randomized clinical study.” 

Aubrey: This is why when people are like, I read a study and it was called the cancer is caused by plastic or whatever, like, no, it wasn't. 

Michael: It's basically, don't read me, don’t read me. 

This is a bizarre take. This is an article in a scientific journal. It is actually quite straight-forwardly titled for a scientific paper. I mean, for goodness sake, it was published in a journal titled, “The Journal of Steroid Biochemistry and Molecular Biology.” It’s not supposed to be for lay people. That’s not the point. I don’t know why Michael and Aubrey are so intent on putting down scientists. 

Michael: So, this is a study of 76 patients in Spain. At the end of the 50 patients treated with vitamin D, only one of them, so 2% ends up in the ICU. Whereas in the control group 50% ended up in the ICU. So, 2% versus 50%. It's like, holy shit, this is a massive effect. I was looking around because I was interested in the sort of, the spread of this paper, and I was looking around at, like, how was this framed at the time? And I found this website called rootclaim, which, like, the Google Doc has the aesthetics of a like, we're just like, nonideological, just objective people looking at data and trying to present you the data. Like this kind of explainer website. It really looks like something very credible. This study obviously, huge reduction death rate from vitamin D. 

Very unclear why Michael is pulling up an incredibly obscure website here that scientists are not using. He acts like he’s the first one to determine that this website is sketchy when he says, “So, I don't want to dunk on this random fucking website…” but it’s also just not an important website? Why is he bringing this up? What does it contribute to the conversation except that it supports his magnification of these debates? Again, the vast majority of the scientific community is not engaging with (or probably even aware of) this website. 

Michael: ...It's like, this is a very small study. It's a study where they give people three different treatments. And so, you can't really say that vitamin D does anything because they're also getting other treatments. The size of the effect is way too big, like suspiciously big and yet there's this massive hype cycle. 

Here’s what Michael gets right: this is a very small study and the effect size is huge. Here’s what Michael gets wrong: both arms are getting the same treatment except for vitamin D. That is how trials work. You can have multiple treatments given in tandem as long as there is one that is different across arms, as in this case. Say, for example, that you and I went out and got dinner together. We share a cheese pizza and you also get a milkshake. Then you get food poisoning. Michael’s implication here is that we wouldn’t know it was the milkshake that caused the food poisoning, because you also ate pizza. Can you see how that logic fails?

Aubrey: That's what scientific studies usually say outright is-

Michael: Exactly.

Aubrey: -further investigation is required, especially with something that matters this much. 

It literally says in the abstract: “Calcifediol seems to be able to reduce severity of the disease, but larger trials with groups properly matched will be required to show a definitive answer.”

Michael: So, this is bouncing around. This doesn't really explode. I think it's very telling that the vitamin D truth or shit really blows up in spring of 2021, which is right when the vaccine comes out. I found this really fascinating study called why were Twitter users obsessed with vitamin D during the first year of the pandemic? Fucking great title. I love it. Straight to the point. That's what my study's about. They look at all these tweets and they include some excerpts of actual tweets that were going around at this time. So, one of them says, actually, let me send this to you because I like it when you do like weird person voice. Do your QAnon voice.

Again, this article is a preprint. The reason Michael likes this title is because it isn’t scientific… It’s just as straightforward as the previous one. But I digress. 

Michael: We're are not going to belabor it, but basically, the consensus is that vitamin D does not prevent or cure COVID. The twist, the big reveal of the episode though, is that vitamin D supplements probably don't work for anything else… So, this whole thing, vitamin D, osteoporosis, like, you hear all these bone fractures and it prevents heart attacks. It makes you live longer, whatever. A lot of that is based on very small studies from the 1980s that essentially researchers have been trying to replicate ever since with larger studies. And they don't replicate. Study after study after study has been coming out for like decades now. Eventually, there's an International Organization of Medicine Panel 2011, they put together a 1100-page report on, like, vitamin D and its links to all of these other conditions. This is from a New York Times article. It concluded that the vast majority of Americans get plenty of the hormone naturally and advise doctors to test only patients at high risk of certain disorders such as osteoporosis. 

So, this thing where everybody needs to be supplementing with vitamin D, everybody needs to be testing for vitamin D. It does not appear to be the case. And there's been tons of other randomized control trials. There's one in 2018 that finds no evidence that it prevents heart attack or cancer. There's another one in 2019 that says it has no effect on cancer. There's a meta-analysis in 2022 that finds no effect. There's eventually an editorial in the Journal of the American Medical Association that is about this latest study. I hate these, Aubrey. It's called “VITAL.” That's the acronym. But it's VITamin D and omegA-3 triaL. So, it's the last letter of omega-3. 

This is not the revelation Michael is suggesting it is. Case in point: the 2010, The Institute of Medicine report (erroneously referred to by Michael as the “International Organization of Medicine” which is not a thing, and said to be from 2011 instead of 2010 - again, such a little thing that would have been so easy to just Google to get right…). What’s bizarre is that Michael is suggesting that this phenomenon, promoted by non-scientists (i.e., Dr. Oz and Oprah) is somehow bad science. Again, this is just an example of the media misrepresenting science. The truth is that vitamin D deficiency is dangerous and linked to a host of health conditions, but supplementation when you already have sufficient vitamin D isn’t necessary. That’s just common sense, though! Epidemiologists have been saying this about multivitamins for ages. 

Michael: So, there's eventually this editorial in the Journal of the American Medical Association called VITAL findings. The findings from this VDOT trial, a decisive verdict on vitamin D supplementation. It says, what are the implications of VITAL? Fact that vitamin D had no effect on fractures should put to rest any notion of an important benefit of vitamin D alone to prevent fractures in the larger population. Adding those findings to previous reports from VITAL and other trials showing the lack of an effect for preventing numerous conditions suggests that providers should stop screening for vitamin D levels or recommending vitamin D supplements. People should stop taking vitamin D supplements to prevent major diseases or extend life. This has gone under the radar. I didn't really know this. I do take a vitamin D supplement. 

So as I mentioned above, it’s been common knowledge in the health/medicine/science community for decades that vitamins are unnecessary unless you have a true deficiency. It appears that Michael is projecting his own naivete onto the general public.  

Aubrey: Boy, oh, boy. As you were walking that through, I was like, this is so similar to the arc of calories in, calories out. Which is that Wishnofsky paper that we talked about that came out in the 50s, that was like this many calories equals a pound of fat. And then people started studying it. They were like, “It's way more complicated than that.” And it lives on in people's minds as like an old tried and true saying. Like, people really continue to believe it to their core. It's fascinating. 

Not at all. Again, calories in, calories out (referred to as ‘energy balance’ in research) is an established truth and is a law of thermodynamics. The ‘3500 cals= 1 lb of fat’ concept was indeed oversimplified, and fat loss/maintenance of that loss is indeed complicated. Scientists know this. That does not negate the fundamental law of physics that energy balance drives body composition changes. 

And again, this vitamin D thing came from Dr. Oz and Oprah. It did not originate from the scientific community. 

---

Conclusion

There’s not much more to say in conclusion that I haven’t said in my previous posts. It is becoming abundantly clear that Aubrey and Michael are creating and promoting content without regard for journalistic integrity. I have reached out to them multiple times, but they have yet to respond, and they have yet to improve the quality and accuracy of their episodes. I think my comments above are pretty self-explanatory, but I truly don’t understand why they don’t hire a fact-checker. In the Iron Culture podcast episode I suggested earlier, the hosts say, “If you have a substantial amount of income for a show that is purporting to accurately communicate science to help people, why can’t you pay a few PhD students in a few different realms who are well-trusted a meager amount of money that could still potentially really fit into your profit margin, I suspect, to do the due diligence to just make sure you have a lower probability of getting it wrong rather than what it seems like here, maybe just being one of those metrics who just read the abstract, before you put it out on a tweet that hits 4.9 million people.” While this is directed at Andrew Huberman, the sentiment applies to MP, as well.