There has been much discussion on the interwebs about the Oregon Medicaid Study, where researchers measured the physical and mental health of 6387 randomly selected poor, non-elderly adult Oregon Medicaid recipients, and compared them to a demographically matched control group of 5842 not selected to receive Medicaid. The particular finding raising alot of eyebrows? “Medicaid coverage generated no significant improvements in measured physical health outcomes” (blood pressure, blood sugar and cholesterol levels.)
However, the study found a substantial improvement in the financial well-being of Medicaid recipients, almost entirely preventing catastrophic out-of-pocket costs. Furthermore, those in the Medicaid group showed substantial improvement in measurements of mental health, in particular depression, compared to the non-Medicaid group. Those in the Medicaid coverage group also significantly increased the probability of a diagnosis of diabetes: a very valuable gain of information for patients and health professsionals, and there were improvements in medication adherence, perception of access to services, and other positives.
The finding of “no significant effect of Medicaid coverage on the prevalence or diagnosis of hypertension or high cholesterol levels” is of course instantly controversial. Because this randomized, controlled study has a quite respectable sample size of randomly chosen new Medicaid beneficiaries compared to matched non-recipients, there were rapid reactions that claimed this bolsters the idea that Medicaid expansion costs too much but does too little:
From Chris Conover at Forbes.com:
These modest gains in health status and financial peace of mind came at a cost. Annual spending was $1,172 (35%) higher for those on Medicaid. Worth noting as well is that these modest improvements in access, use, quality, mental health and financial protection produced no significant difference in self-reported happiness. Despite being uninsured for at least six months, 74.9% of the control group reported being very happy or pretty happy compared to 76.1% of those on Medicaid.
So it all comes down to value for money. Are these gains worth spending roughly $1,200 apiece to give uninsured adults below poverty Medicaid coverage? Every reader will have their own opinion on this matter. But most would agree it’s not worth spending an average of $1,172 if all Medicaid achieves is a reduction in average out-of-pocket spending by only $215. In that case, it would be far cheaper to simply reimburse individuals for expenditures above some agreed-upon catastrophic threshold.
From Megan McCardle at Daily Beast:
You can squint hard at the data and say, well, sure, the effects weren’t statistically significant, but there was some improvement! Much such squinting has been going on. But if there had been a slight, not-statistically-significant decline in the health of the Medicaid participants, I’m skeptical that many–or any–of our squinters would have been touting the probative power of those sorts of small effects. As someone I was talking to earlier noted, “It’s got huge confidence intervals” is not normally the sort of thing you hear when arguing that a study supports your thesis. Our intuitions about health care, not the data, are doing a lot of heavy lifting here.
When you do an RCT with more than 12,000 people in it, and your defense of your hypothesis is that maybe the study just didn’t have enough power, what you’re actually saying is “the beneficial effects are probably pretty small”. Note that we’re talking about a study the size of a pretty good Phase III trial for Lipitor, Caduet, or Avandia–some of the leading new drugs for treating high cholesterol, hypertension, and diabetes. Of course, to be fair, those trials enroll only people with the disease they’re targeting, so you should get more statistical power–but then, to also be fair, many of those studies have many fewer than 12,000 participants and still achieve statistical significance.
And as Katherine Baicker, a lead researcher on the Oregon study, noted back in 2011, “people who signed up are pretty sick”. Yet the study failed to find statistically significant improvement on the three targets associated with the most common chronic diseases. This, mind you, is the stuff that we’re very good at treating, and which we’re pretty sure has a direct and beneficial effect on health.
One of the more interesting counterarguments comes from Austin Frakt in the New York Times:
While the latest study failed to show any statistically significant effects of Medicaid on physical health, there’s a simple explanation: the sample was far too small for it to be able to do so.
The larger body of work on the health effects of Medicaid expansions throughout the country shows them to be beneficial, particularly for vulnerable populations including those with severe illnesses.
Would the program work as well or better if it was converted to catastrophic-only coverage? It’s hard to be confident it would.
The issue of the Oregon Medicaid Study’s sample size and statistical power (the sensitivity, or probability that it will reject a false null hypothesis, in this case, that Medicaid had no effect on blood sugar) gets a more detailed treatment with a Frakt blog piece, where he quotes Mother Jones’ Kevin Drum:
Let’s do the math. In the Oregon study, 5.1 percent of the people in the control group had elevated GH [glycated hemoglobin, aka A1C, or colloquially, blood sugar] levels. Now let’s take a look at the treatment group. It started out with about 6,000 people who were offered Medicaid. Of that, 1,500 actually signed up. If you figure that 5.1 percent of them started out with elevated GH levels, that’s about 80 people. A 20 percent reduction would be 16 people.
So here’s the question: if the researchers ended up finding the result they hoped for (i.e., a reduction of 16 people with elevated GH levels), is there any chance that this result would be statistically significant? […] The answer is almost certainly no. It’s just too small a number.
Frakt claims the study “wasn’t even designed to detect a 20% decrease in elevated GH levels, let alone 10%, 5%, or 1%.” He goes on to estimate how large the study would need to be to have the statistical power to be able to reject the null hypothesis, i.e. whether Medicaid had no effect on blood sugar:
Let me now state where we are. With respect to the statistically insignificant physical health measures in the study, we now know they were very underpowered. The sample was too small even for much larger effects. This renders them statistically uninformative in general and, in particular, uninformative about whether or how much Medicaid improves physical health. Uninformative means just that. No new information. No resetting of priors is justified on this question.
We also know, from the authors’ discussion and from Aaron’s posts, that the results include changes in blood sugar and blood pressure that are not unreasonable to have expected clinically. Thus, the results — or these two anyway, but I suspect it generalizes — are not clinically informative either. Again, no resetting of priors is warranted.
Given this, for the physical health measures only, I don’t understand the rush I’ve noticed in people updating what they expected Medicaid could do. These results really shouldn’t do that if they are, as I’ve said, uninformative both clinically and statistically. How did people make these judgements the day after the study was published? It’s taken me and Aaron almost two weeks to chase things down. I think it is time for people to take another look at what this study is saying, at their own priors, and, yes, at their own biases.
What I think we’re seeing is a re-expression of everyone’s priors. This study is an opportunity to do that, but it doesn’t and shouldn’t change what they are. The claims that people should be changing sides from pro- to/from anti-Medicaid expansion just make no sense based on the physical health measures in this study.
Meanwhile, yes, this study reconfirms some large financial and mental health benefits that we knew about from last year’s paper from the group. I’m not sure that’s a prior-updating event either.
This was (is) an excellent study done by a smart and capable team of investigators. The results, to the extent they are meaningful, should be viewed as among the most credible possible within the context of the study (in/around Portland, over 2 years, Medicaid circa 2009). And yet, much too much is being made of the set of results that just don’t tell us anything new.
Chris Conover from Forbes gets into the fray with Frakt, and cites the RAND Health Insurance Study from the mid-80’s:
You keep harping on the diabetes result, yet lack of statistical power surely cannot explain the failure to find a significant result for high blood pressure, especially in light of the RAND HIE findings, which I explain here: http://www.forbes.com/sites/chrisconover/2013/05/07/does-the-oregon-health-study-show-that-people-are-better-off-with-only-catastrophic-coverage/
For low income non-elderly adults, the OHS had a far bigger sample than the HIE, yet the HIE was able to demonstrate a statistically significant improvement in HBP in the free care group compared to those in cost-sharing plans. The OHS, with a much bigger sample, failed to demonstrate Medicaid has an impact on HBP compared to people with no health insurance whatsoever!
The commentary and back and forth among McArdle, Drum, Frakt, Conover and others is enlightening. We are seeing genuine critique and engagement, all too rare these days. What does it all mean? The parameters of disagreement seem to have narrowed. Medicaid seems to produce measurable health benefits, and patients are spared financial hardship, but it is not apparent that there are demonstrable improvements in particular areas of greatest concern (i.e diabetes and the metabolic syndrome). If there had been obvious, pronounced benefits for Medicaid enrollees in terms of blood sugar levels, a randomized controlled trial with 6387 randomly selected poor, non-elderly adult Oregon Medicaid recipients, and a control group of 5842 should have picked it up.
This is just one state, and just one study. To be continued.
Health Informatics Web 