counter statistics

Monday, December 15, 2008

Teen Challenge's So Called Study

Maia Szalavitz takes apart Teen Challenge's claim that a study has shown them to be "74% effective."

When Minnesota Teen Challenge (MNTC) responded to my recent blog entry about their anti-drug program, they cited a "study" to back their claims of being an effective treatment for addiction.

What this paper actually shows is how easy it is produce good looking numbers. In that way, it's actually quite instructive for anyone who wants to understand addiction research--or wants to avoid being taken in by exaggerated outcome data.

Even more intriguingly, MNTC failed to mention that the success rate they proudly cited is for adults--but buried in their own study is a much more dismal picture for teenagers.

Their uncontrolled research includes the classic statistical ploys used by drug programs for decades to inflate success rates. At first glance, a claim of 74% of graduates abstinent without relapse for six months sounds pretty good. OK, it's only six months--but often, if you can make it six months at a time without relapsing, you are doing pretty well.

Let's look a little closer at that number and how it was generated, however. Start with the fact that MNTC participants are not just "addicts off the street." They are seeking treatment--either because they have to in order to avoid prison or because they have decided they want to stop using.

Most have been through a detox program to help with withdrawal--and many will have dropped out before completing that. MNTC participants have also consented to attend a highly religious rehab--or had their parents consent for them. This suggests that we have already eliminated many of the addicts who aren't motivated to recover before they even set foot in the door.

Because there is no control group, all of those facts already mean that any success in the program we see could be due to pre-existing motivation: not to anything special about the rehab. Only with a control group of similar people who get no treatment or attend a different rehab can we really tell what works and what doesn't.

Read the whole thing. If there is no control group in this study, then the "74% effectiveness rate" doesn't mean anything.