Advice is everywhere, but it’s rare to know what benefit you’re supposed to get from following it.
Say you’re a smoker. If I tell you not to smoke, you’ll probably ignore me because you think (correctly) that I care less than you do about the cravings you’d suffer if you quit. But if someone reminds you that each cigarette takes 10 minutes off your life, you’ll probably consider that.
Or say you hate vegetables. You know they’re good for you, but how good, enough to outweigh the horror of eating okra?
The solution is “effective selfishness”—advice that comes with effect sizes.
Effect sizes are hard
To know what running does, you should take thousands of people, tell half to run, wait years, make sure people follow their instructions, and then see how healthy everyone is at the end. That’s not easy, and even if you did that, you still have to worry: Did the running itself make people healthier? Or did people get more sunlight while outside or make friends at the park or get a placebo effect from the belief that running is healthy?
The easiest place to find effect sizes should be small-molecule drugs: There’s no interaction with lifestyle, you can control for placebo effects with placebo pills, and these experiments are what the medical system, in its grace, likes to spend money on.
And in terms of small-molecule drugs, what should be easiest? I chose statins, the drugs that 200 million people around the world take to control cholesterol. These are old, cheap, commonly used, and thought to have huge effects, so we have lots of big studies.
Aside: Statins have an interesting mechanism of action: HMG-CoA reductase is an enzyme that takes HMG-CoA and creates mevalonic acid, which is needed to make cholesterol. Statins are molecules that are structurally similar to HMG-CoA, meaning they sort of get stuck on the HMG-CoA reductase enzyme where HMG-CoA is supposed to go, meaning the enzyme can’t make mevalonic acid, which bottlenecks the whole cholesterol pipeline.
Anyway, there are many meta-analyses for statins out there, but they typically work on risk ratios, i.e. the relative chance that someone will suffer a heart attack or die during a given period. These are great, but they aren’t particularly actionable: If I don’t have a great sense for what my odds of dying are (which I don’t) then it’s hard to say how important it is if those odds get multiplied by a given constant.
So, being a fan of simple-minded “number of days of life” effect sizes, I was excited to find Kristensen et al.’s paper, The effect of statins on average survival in randomised trials, an analysis of end point postponement. They try to do exactly what we want:
To the best of our knowledge, statins have not been systematically assessed in an outcome postponement model. We identified statin trial reports that provided all-cause survival curves for treated and untreated, and calculated the average postponement of death as represented by the area between the survival curves.
Kristensen et al. do a meta-analysis of 11 studies. Let’s take a quick look at a couple of them to get a feel for the data. (We’ll need the intuition later.)
The WOSCOPS study (1995) took 6595 Scottish men aged 45 to 64 who had high cholesterol (total cholesterol at least 252 mg/dL) and no history of heart attacks. They were given either 40 mg/d of pravastatin or placebo. At the end of the study, 4.1% of people in the placebo group had died of any cause while only 3.2% of the pravastatin group had died. The difference has a vexing p-value of 0.051.
The LIPID study (1998) took 9014 patients aged 31 to 87 in Australia and New Zealand who had a history of either a heart attack or chest pain, as well as moderately high cholesterol (total cholesterol between 155 and 171 mg/dL). They were given either 40 mg/d of pravastatin or placebo. At the end of the study, 14.1% of the control group had died as compared to 11.0% of the treatment group. The difference is very significant.
The JUPITER study (2008) took 17,802 people from around the world who had moderate cholesterol (LDL-C ≤ 130 mg/dL) and elevated C-reactive protein levels (≥ 2.0 mg/L). Men needed to be at least 50, while women needed to be at least 60. They were were given either 20 mg/d rosuvastatin or placebo. At the end of the study, 1.19% of subjects in the placebo group had died (of any cause), while 0.96% of the rosuvastatin group had died.
There are a couple of things about the JUPITER study that are slightly suspicious.
For one thing, it was financed by AstraZeneca who sell rosuvastatin under the brand name of Crestor. Actually, it wasn’t just financed, AstraZeneca also collected the data and monitored the sites, but was firewalled from influencing the analysis or manuscript.
For another, these mortality rates are weirdly low. For example, mortality in the US for people aged 55-64 is around 0.88% per year, and for people aged 65-74 is it 1.78%. Yet the overall mortality in the placebo group was only 1.19% even after multiple years? It’s possibly explained by the fact that they excluded people with high cholesterol but it’s still weird and apparently, I’m not the first person to notice that.)
Looking at these studies, these results seem amazing—a simple pill can reduce the entire risk of dying, from any cause by around 25%! It’s hard to imagine asking for more.
And yet. Let’s get back to Kristensen et al.’s meta-review. Remember, they wanted to calculate how many days of life are saved by taking statins. Here’s what they calculate for the above studies:
|study||Days of life|
There’s a benefit of… only a few weeks? How can a magic pill that erases 25% of all deaths translate into such small numbers?
First off, let’s look at their calculations. Here’s part of their analysis section, which I include just because it’s endearing to see “we used MS Paint” translated into High Academic.
Basically, if you take the survival curves and measure the areas under them, you get the average number of days people in the treatment and control groups were alive during the experiment. If you take the difference, you extra days of life from statins.
So how to reconcile these small numbers with the large-seeming effects from before?
The main issue is that their analysis only looks at extra days of life during the study itself. Take the WOSCOPS study. When it was over, 4.1% of the control group were dead, as compared to 3.2% of the treatment group. Let’s divide people into three groups:
- People who would survive regardless of statins (95.9%)
- People who would die regardless of statins (3.2%)
- People who would survive with statins but die without (0.9%)
The analysis above doesn’t account for any extra life that the 0.9% of people in the last group rack up after the study finished. It’s as if they all died the day the study was over.
Let’s do a very rough estimate. The mean age of subjects at the start of the study was 55 years, it lasted five years, and the life expectancy for a 60-year old in Scotland in 1994-1996 was 17.2 years. If we assume that the people in the last group have standard life expectancies, then this means that there is an average of
17.2 × 365 × 0.009 = 56 days
that’s not being accounted for.
(If you’re a stats nerd worried about the variance in the age distribution, stay calm: You can check that the life expectancy tables are reasonably symmetric for ages near 60, so if the age distribution is also symmetric, it all cancels out.)
Now, assuming that group will live for 17.2 years on average might be too much—these are people who survived with statins but would have died without, so they are probably in somewhat below-average health.
But there’s another factor: The future outcomes of the 95.9% of people who would have survived the study regardless of statins. We don’t have data, but I feel comfortable saying that the people in this group have better prospects if they’ve been taking statins. I mean, statins worked so far, won’t they keep working? Or just look at the figure from above again—the curves look like they are still bending away from each other at the end, i.e. differences seem to still be accelerating.
So we have one way in which our calculation is too aggressive and one way it’s too conservative. With no justification at all, I’ll assume these cancel out, so the adjustment is still 56 days, but with lots of model uncertainty.
We can repeat this analysis with each of the other studies.
- Median age at start: 55 years
- Marginal population (survive with statins but not without): 0.9%
- Length of study: 5 years
- Life expectancy for a 60 year old Scott in 1995: 16.2 years
- Adjustment: 17.2 × 365 × 0.009 = 56 days.
- Mean age at start: 63.1 years
- Marginal population: 0.5%
- Study length: ~3.5 years
- Life expectancy of 67 year old Swede in 2003: 17.18 years
- Adjustment: 17.18 × 365 × 0.005 = 31.35 days
- Mean age at start: 61.65 years
- Marginal population: 1.5%
- Study length: 4.75 years
- Life expectancy of 66 years old United Kingdom human in 2004: 17.41 years
- Adjustment: 17.41 × 365 × 0.015 = 95.32 days
- Median age at start: 66 years
- Marginal population: 0.23%
- Length of study: Varies, people signed up through most of 2003-2006, and the analysis considers all events through when it was terminated on March 29, 2008. Let’s call it 3 years?
- Life expectancy of a 66 year old American in 2008 (since I can’t tell where the subjects in this study came from): 18.16 years
- Adjustment: 18.16 × 365 × 0.0023 = 15.24 days
- Mean age at start: 59 years
- Marginal population: 3.6%
- Study length: 5.8 years
- Life expectancy of 65 year old Swede in 1994: 17.99 years
- Adjustment: 17.99 × 365 × .036 = 236.39 days
- Median age at start: 62 years
- Marginal population: 3.1%
- Length of study: 6.1 years
- Life expectancy for a 68 year old Australian in 1998: 16.24 years
- Adjustment: 16.24 × 365 × 0.031 = 184 days.
Putting this together, we finally get these estimates:
|study||Original days||Adjusted days||Length of study (years)|
The increases range from a factor of 3 to a factor of 16.
The major trend in the above table is that larger studies tend to find larger effects, both in terms of the original estimate and my very rough adjustment. For this reason, if I had to guess, I’d guess that my adjustments are still too low, i.e. the benefits of statins are larger than the above would suggest.
There's a bunch of other studies that I didn't include here, either because they weren't placebo-controlled or because the study population was already having particular medical issues.
There’s a bunch of studies I didn’t use. One is ALLHAT-LLT (2002), which 10,355 North Americans who were 55 or older with moderately high cholesterol and triglycerides that weren’t too high (LDL-C between 120 and 189 mg/dL and triglycerides less than 350 mg/d). They gave people either 40mg/d pravastatin or usual care.
Here the black line is pravastatin, and the dotted line is usual care. There’s minimal effect, and for much of the study, the usual care group even did a bit better. At the end of the study, a slightly lower fraction of those in the pravastatin group had died (14.9% vs 15.3%), but it’s obviously not significant.
While this at first seems to suggest that statins don’t do anything, there one huge caveat: The “usual care” group kept going to see their doctors, who were free to prescribe them statins, which many did. This means that around 30% of the usual care group also took statins. Arguably, what this study shows is that mindlessly giving everyone the same dose of the same statin works as well as having thousands of doctors examine everyone and apply all of their talents. That’s certainly interesting, but probably beside the point for us.
Two other studies (MEGA, and GISSI-P) also weren’t placebo-controlled. Finally, GISSI-HF had subjects on dialysis and CORONA had subjects with ongoing systolic heart failure. These are all so different that I’m skipping them as well.
Now, I kept every study with a placebo control that targeted a “normal” population, which I think is the most sensible approach. But I still feel compelled to mention that these rules exclude 5 of the 6 studies with the smallest (or even negative) effects. That’s particularly strange with non-placebo-controlled studies, which you’d expect to produce a larger effect size, not a smaller one.
If your doctor says you should take statins, then you should probably take statins. I think the previous analysis that said you only get a few weeks is too conservative, and a better estimate is a few months.
(By the way, Kristensen et al. are well aware that their analysis might be conservative—none of this is a criticism of them!)
Now, even with my adjustments, you might find these effects to be surprisingly small. If the above table makes you think statins aren’t worth the trouble, let me make a couple of points:
- Try inversion: Suppose I offered you a lifetime supply of delicious cookies, one per day, except if you eat the cookies, you’ll die a few months earlier. Would you take them? (For this to work, you’re supposed to find the pleasure of the cookies equivalent to the inconvenience of taking statins.)
- Lifestyle interventions that buy you a few months are a big deal! If you want to have a longer, healthier life, it will probably come from a combination of things, each of which has a modest effect. “A few months here, a few months there, pretty soon you’re talking about real money.”
Still—and despite how dismally non-contrarian all this is—please don’t listen to me when making medical decisions!