Mining Healthcare Data : A Modern Rumpelstilskin Story

Via Megan McArdle: the New York times and the Washington Post are reporting on recent problems stemming from the Obama Administration’s Healthcare-data project. Apparently data analysts studying health impacts of new programs are not controlling their experimental samples. Whereas ideally the government’s analysis would be the basis for crafting intelligent policy, the New York Time’s description calls into question the robustness of the research being conducted.

“The studies that are regarded as the most reliable randomly assign people or institutions to participate in a program or to go on as usual, and then compare outcomes for the two groups to see if the intervention had an effect.

Instead, the Innovation Center has so far mostly undertaken demonstration projects; about 40 of them are now underway. Those projects test an idea, like a new payment system that might encourage better medical care — with all of a study’s participants, and then rely on mathematical modeling to judge the results.”

The superficial approach described above is odd because it seemingly flies in the face of conventional approaches statistical modeling. For those not familiar, establishing a randomized control is essential to getting results that don’t just confirm the hypothesis is being tested. You can see this problem in the infamous Israeli Air Force Study(a really informative overview of this concept can be found on YouTube), and it’s been a long standing statistical understanding that, when possible, randomized control samples are always preferable.

So why do government analysts feel so confident that they can dispense with what has, until recently, been an essential feature in any statistical experiment? Well because they’ve got great data-mining technology!  Here, the word “mathematical modeling” does a lot of work in obscuring the real methods that the government is using. Mathematical modeling can really mean anything, and ironically the NYT’s link on this description is broken.

Megan McArdle, has a good take on the possible sources of the mistake: sloppy thinking on the part of federal bureaucrats. Says McArdle:

Gold’s article implies that the administration is looking at gross savings — which is to say, it’s just reporting the amount of money saved by the accountable-care organizations that ended up on the positive side of the ledger, even though this is less than half the total. Statisticians have a term for this: the Texas sharpshooter fallacy…..

Perhaps, I may be even more cynical than McArdle, but my take is somewhat different.

Given that the administration has been unable to produce evidence of healthcare savings from increased coverage, it is fair to say that the president is feeling pressure to come up with some statistical result that will make costs appear more reasonable  (at least ahead of the next CBO estimate). Moreover, without speculating too much as to the overall structure of bureaucratic management, I don’t think it is unlikely that individual analysts are also feeling the pressure to deliver “good” results, especially with all of these cool new “big data” tools so prominently featured in the news.

The result is predictable: a sort of magical thinking arises where data-mining and complex models become panacea for turning poorly conducted statistical tests into predictive models showing large savings from new “innovative” approaches to delivering healthcare. Of course the results are all confirmation bias, but who’s going to look a gift horse in the mouth? Certainly not an administration desperate for good news on the healthcare front.

Now admittedly, I have no inside information, but if this kind of sloppy analysis is indeed going on then it is certainly a cause for concern. The one-sided use of over-optimistic healthcare predictions could lead the CBO to perennial underestimate the cost of supporting programs like Medicare in their current state. This in turn could ultimately doom these program’s long-term solvency (not to mention the long term solvency of the country) since politicians are all too willing to forgo necessary reform in the light of CBO reports that tell them healthcare costs will come down on their own accord.

But ultimately this problem is not political. It stems from a cultural approach to data analysis that is far too prevalent in industry and in government. I like to think of it as a modern day Rupelstilskin story. What do we have? Reams of uncontrolled data. What Do We Want? Optimistic predictive results. With this point of view, it’s tempting to simply lock analysts in a room and ask them to build mathematical models until they finally manage to spin the straw data into golden predictive models like the miller’s daughter from the aforementioned fairy tale.

But just as in the fairytale, when we force someone to spin straw into gold, it shouldn’t be surprising when magical methods play a large role in their process. Moreover, in the case of the government’s own analysis the Rupelstilskin metaphor can be taken yet further. For in trusting their magic numbers, our current leaders may have put the next generation on the line for the results.

Culture, Speech Codes and Implicit Bias

I recently came across a brialliant keynote by Megan McArdle to the libertarian-leaning FIRE organization about the increasing prevalence of campus speech codes. What follows is a more philosophical examination of free expression beyond the typical “administrators who suppress student speech are bad”, I recommend it for anyone who has half an hour.

Like most people who find themselves on the “wrong” side of what college administrators typically consider correct, I’ve had an interest in the growing attempts by campuses to lock down expression in the name of sensitivity. But I think Megan has nailed why these codes emerged in the first place (and no, it’s not leftist academics secret desire to re-institute Stalinism). In most cases, it seems that the speech codes are an attempt to replace the missing cultural commonality that was lost when American Universities became multicultural in the late 70s and early 80s. Stringent speech and behavior codes, in effect, were an attempt to run a multicultural student body under a mono-cultural system of support this time with progressive empathy filling in for the old WASP sense of decor. Basically, it’s a way to avoid the problems, and benefits, of living in a multicultural campus. No one wants to have uncomfortable interactions and the speech codes continue the illusion that none are actually necessary.

Of all the bloggers writing today, Megan McArdle seems to have the strongest understanding of how institutions re-enforce their own implicit bias and unanimity, not with ill-will, but through the implicit ways humans prefer to interact with others of alike mind. She’s right in identifying this force as the reason why university sociology departments are overwhelmingly non-conservative and university science departments are overwhelming non-black. The mechanism is the same. I think many people of the left acknowledge just this kind of effect when it comes to structural racism, sexism, and hetrosexim. But they are resistant to acknowledge this as a general flaw in all human organizations, particularly organizations that they prefer. Hence, I always get the sense that the liberal solution to structural racism is to ensure that the “right” people are put in power.

But at any rate, regardless of who is correct, this is a conversation worth having, though rest assured it won’t be a very a sensitive or comfortable one.

 

Yes Ralphie, there is Social Utility in Christmas Materialism

As predictable as Christmas TV-specials, Black Friday sales, and candy cane-themed mall directions, one can expect the perennial gripes from the chattering class over the increasing materialism in the holiday season. Over the years I’ve gown a bit tired of this complaint so it was refreshing to hear an intelligent push back from the ever-sharp Megan McArdle at the Bloomberg view. It turns out there is utility in giving gifts rather than purchasing things for yourselves. The holiday season is economically vindicated.

“…buying gifts for someone else actually makes you happier than spending it on yourself. They gave people money, and told them either to get something for someone else, or spend it on themselves. The people who spent it on someone else reported being much happier with the experience than the folks who bought themselves something. It seems it really is better to give than to receive.”

Read the whole thing. Personally, I feel this makes absolute sense (and should for any one who has ever shopped on the holidays).  Leaving aside the obvious sentimentality of giving and receiving gifts with friends and family, the entire endeavor has the combined fun of solving a mystery and being part of a conspiracy. Moreover, it’s nice to be forced to figure out the various things members of our families might want. As someone who lives far away from my hometown, I know much more about my family and old friends than I would otherwise.

I know, I know, buying things for people isn’t what Christmas is about. But we live in a culture that is defined by material things. Saying that people should spend quality time instead of giving is well and good. But making such gestures the norm will involve a fundamental reform of a culture where people line up for hours to get the newest phone upgrade. So for the time being being at least, we can enjoy our materialist Christmas, in spirit at least.