Evaluations of Little Value in PHI

As the popularity of comparative cost-effectiveness research increases in reactive medical care, it is understandable that proactive health management is getting similar attention. Two recent articles point out some of the problems with such research when applied to wellness and chronic disease management, however, though the tendencies involved are horribly common.

Echoing the consistent conclusions of government-sponsored studies of disease management, a macro-analysis of studies conducted by the Center for Studying Health System Change concluded that: “…return on investment for wellness initiatives is uncertain…”. [“Wellness Programs Don’t Necessarily Improve Health or Lower Costs” Wall Street Journal Health Blog July 29, 2010 (blogs.wsj.com/health)] While most such studies are limited in their scope to examining reductions in sickness care costs as the only source of ROI, the idea of attempting to reach general conclusions about proactive health efforts is absurd in the first place.

Sickness care studies require rigid control studies, where the medical intervention is the same for all patients in the intervention group, and a control group ensures that the only difference between the intervention group and the control group is that one got the intervention and the other didn’t. By contrast, when comparing wellness interventions, the intervention is almost never the same for all, or even a majority of the people in the intervention group. Wellness interventions vary all over the map, with respect to what kind of health goals are involved, what methods for coaching are applied, etc. There is no real possibility of reaching a general conclusion about “wellness” from studies involving widely varying interventions, anymore than there would be if a similar attempt were made to evaluate the ROI from “medical care” in general.

Instead of wasting money, time and media attention on general conclusions based on the small proportion of research meeting clinical standards on wellness interventions, it would be far better to use more widely-focused wellness studies to identify precisely which among the host of interventions being used actually do work, which are most cost-effective, and therefore, which should be emulated. A similar approach should probably also be used with “complementary and alternative medicine” (CAM), which includes such an incredible array of non-medical interventions as to make the idea of reaching general conclusions about it ridiculous from the outset.

Another all-too-common silliness applies to media and industry reports about particular types of health plans. In another recent article, the efficacy of Medicare Advantage health plans was reported with respect to reducing patient re-admissions, a goal of both payers and hospitals, as threats of non-payment for presumably avoidable re-admissions make them worth avoiding. [J. Lubell “Advantage Plans Helping to Reduce Readmissions, AHIP Says” ModernHealthcare.com July 28, 2010]

The conclusion that Advantage plans reduce re-admissions was apparently based on cross-sectional comparisons of rates between populations that were enrolled in such plans compared to populations enrolled in traditional fee-for-service plans. While such comparisons are always of some interest, they absolutely do not show that Advantage plans cause readmissions to go down. They merely describe different rates in the different populations, e.g. 16.7% in Advantage vs. 20.5% in Texas. Unless analysis shows that patients in Advantage plans had lower re-admission rates after they enrolled in such a plan, compared to what their rates had been before they enrolled, plus that there were no other possible causes of the lower rates likely to have caused such a difference, the comparison of rates between two different population is no more than an interesting phenomenon.

It is understandable that people and organizations with “an agenda” pre-disposing them to find positive, negative, or equivocal results will do so, whatever the data show. I recall an example when I was a doctoral student where the director of a research project reported a conclusion exactly opposite what the data showed, reflecting her strong, though erroneous convictions, rather than the data. When economic self-interests are involved, similar bias is possible. And the media seem to prefer interesting, even startling results when reporting findings to careful analysis of the data underlying such results.

Evaluation should focus on delivering sound and useful conclusions that will advance the causes addressed, rather than, as Mark Twain noted, yielding “…wholesale returns in conjecture for trifling investments in fact.” Far too many evaluations of little or no value are routinely carried out, and unfortunately published.