If you've looked into the controversy regarding the use of ivermectin for treating COVID-19, chances are you've come across links to a site called <a href="http://c19ivermectin.com" rel="nofollow">c19ivermectin.com</a> (or one of its many relatives) that claims to host a regularly updated aggregation of all the latest studies into a single meta-analysis of the effects of the drug. We here at Ars have been asked—by email, in the comments, and via our feedback form—to check out <a href="http://c19ivermectin.com" rel="nofollow">c19ivermectin.com</a>, which purports to provide evidence ivermectin is an effective therapy.

So we have. And we're suggesting you don't, because it's not a reliable source of information.

We want an old drug

Why is anyone talking about ivermectin at all? Early on in the pandemic, before the development of effective vaccines, there was a rush to find treatments that could be rapidly rolled out to the public. As such, researchers focused on testing drugs that were already approved for other problems, since this would lower the regulatory hurdles and safety testing needed. Plenty of existing drugs were tested in cell culture, and a few were trialed in humans.

Remdesivir, a drug that was originally developed for Ebola, was one of the things that came out of this effort; it seems to have a minor effect on COVID-19's duration. But so was hydroxychloroquine, which got excessively hyped but turned out to do nothing.

One of the other drugs that was tested in cultured cells was ivermectin, which was developed as an anti-parasite treatment. It's effective against a number of human diseases and sees widespread use in the veterinary world. At therapeutic concentrations, it specifically binds to proteins found on the nerve cells of parasites and shuts down neural activity. But at higher concentrations, it starts to bind to the equivalent proteins on human nerve cells, causing side effects that can be severe.

One very large human trial appeared to show that ivermectin was effective against COVID-19, but it was later withdrawn following accusations of plagiarism and manipulated data. Since then, a number of small, variable-quality trials have had varying results. An analysis of the state of the field published in late July indicated it wasn't possible to use these studies to determine if ivermectin was helpful or harmful. The situation doesn't seem to have changed dramatically since. Currently, most health authorities indicate that there is no evidence supporting ivermectin's use against COVID-19.

But some doctors are hyping ivermectin anyway, and groups have formed on social media platforms that swap ideas on how to take the drug. That has, unfortunately, caused a number of individuals to turn to veterinary formulations, boosting calls to poison control centers and producing at least one overdose. As a seemingly scientific compilation of evidence that ivermectin is effective, <a href="http://c19ivermectin.com" rel="nofollow">c19ivermectin.com</a> appears to be providing some of the justification these people need.

Meta chaos

The site purports to perform a real-time meta-analysis of all the data from human trials of ivermectin in COVID-19 patients. A meta-analysis is a way to aggregate studies that, on their own, don't have enough participants to provide a statistically significant result. While individual trials might enroll only 50 patients and provide ambiguous information, 10 such trials would give you a total population of 500, which could provide a clear effect.

The challenge with doing a meta-analysis is that few research teams use exactly the same protocol—trials may differ in the populations enrolled, the control groups used, the timing and dose of the drug, the outcome measures, and more. So part of performing a robust meta-analysis involves figuring out which studies use equivalent procedures and so can have their participants treated as a single population. The chaotic mixture of protocols used in ivermectin trials is one of the reasons the research community is uncertain about the drug's function.

If we look at the most detailed explanation of the meta-analysis performed at <a href="http://c19ivermectin.com" rel="nofollow">c19ivermectin.com</a>, however, we can see that the people behind the site don't really care about any of that. In fact, they essentially do the opposite by throwing every single trial into their analysis and criticizing other researchers for being selective. "The primary methods used [by other meta-analyses] to avoid reaching statistical significance is exclusion of the majority of the evidence base, and division of the remaining subset," <a href="http://c19ivermectin.com" rel="nofollow">c19ivermectin.com</a> complains. "It is invalid to use partial evidence from a small subset of studies and then claim there is not enough evidence."

Because of this attitude, the trials used in c19ivermectin.com's meta-analysis are all over the map. Sometimes the patients are blinded to the test treatment, while other times they know exactly what they're getting. The dose over four days ranges from 12 milligrams up to 158 milligrams, and the timing of that dosing varies wildly. The age and comorbidities, end point measurements, and more show no consistency among the trials—sometimes those with comorbidities were excluded entirely, other times they weren't.

In many cases, other drugs were administered at the same time (one trial lists ivermectin, azithromycin, and cholecalciferol; another just ivermectin; yet another azithromycin, zinc, vitamin C, vitamin D, and paracetamol). Time since infection varied as well.

In short, there's absolutely no way these studies should be combined into a single analysis. And many of them involve so many additional drugs that it's not clear if they can tell us anything about ivermectin at all.

Who’s in charge here?

In defending their work, the <a href="http://c19ivermectin.com" rel="nofollow">c19ivermectin.com</a> researchers note that they've run analyses in which they exclude different categories of studies, and no matter how the data is sliced, the results for ivermectin are consistently positive. But that's probably what you'd expect when all but five of the 60-plus studies you've looked at show a positive result (as you can see in Figure 1 of the document linked above). No matter how you slice things, most of the results are going to be positive.

Obviously, this high frequency of positive results would be expected if the drug actually worked. But you'd also see it if there's a tendency to publish any positive results, even if the underlying experiments are poorly controlled or suffer other flaws. Is there any indication that the latter might be the case?

Conveniently, the people behind the site seem to have provided it by performing a similar analysis on other purported COVID-19 treatments. Of the 25 other treatments put through this analysis, the meta-analysis indicates 24 are effective COVID-19 treatments. This includes hydroxychloroquine, which we now have pretty definitive evidence is ineffective. A further indication of the problems with this approach is that many of the substances listed in the treatment cocktails above—zinc and vitamins C and D—are listed independently as effective treatments. But hey, what's a few confounding factors when it comes to a lethal disease?

Presumably, the folks behind the website would explain everything working by pointing to the site's FAQ, which says, "We focus on the most effective/promising early treatments." But it's not unreasonable to ask whether their approach has produced much in the way of negative results that they've not put on the website. In fact, asking this question is essential if we're to evaluate this sort of meta-analysis.

Just don't ask the people who performed the analysis. Because they've chosen to remain anonymous, while at the same time claiming extensive scientific expertise.

From the same FAQ:

We are PhD researchers, scientists, people who hope to make a contribution, even if it is only very minor. You can find our research in journals like Science and Nature. We have little interest in adding to our publication lists, being in the news, or being on TV (we have done all of these things before but feel there are more important things in life now).

There's no obvious way to verify their expertise. All of the different domains that provide access to the website (there's a general URL, drug-specific domains, and two different ones for ivermectin) were registered by a large Canadian service provider. All Ars could determine is that it's distinct from the Front Line COVID-19 Critical Care Alliance, a different pro-ivermectin group that uses a US provider.

So, to sum up: the methods the people behind the website are using appear to be badly flawed, and their approach appears to be biased towards producing positive results, even with treatments that are pretty clearly ineffective. They criticize other researchers for sticking with rigorous standards but wall themselves off from criticism for their lack of standards by remaining anonymous.

Approach with caution

None of this is to say that more definitive studies won't eventually determine that ivermectin is effective against COVID-19. But experienced public health authorities have concluded that those studies haven't been done yet and that the studies that have been done don't collectively amount to evidence.

The people behind <a href="http://c19ivermectin.com" rel="nofollow">c19ivermectin.com</a> are attempting to provide a counternarrative to that and are doing do their best to present it in a way that it appears scientific. But their approach raises serious questions about whether it can possibly produce valid information. And the fact that it produces spurious results with other chemicals suggests that it doesn't.

There's an old chestnut that the folks behind research don't matter and the data should speak for itself. But the methods used to analyze data typically matter just as much, and methods often do not speak for themselves. And, if the methods and data produce an answer that goes against what most other scientists have concluded, then it's often worth identifying any external motivations that might drive the disagreement. The people behind <a href="http://c19ivermectin.com" rel="nofollow">c19ivermectin.com</a> aren't doing anybody—including themselves—any favors by remaining anonymous.