OP-ED: Evidence-based Practices Aren’t the Only Tool in the Shed

Print More

Amanda PetteruitiOver the last 15 years, juvenile justice advocates fought hard to convince policymakers and government officials that the best way to help youth succeed and improve public safety is to keep them out of secure confinement. To keep youth out of confinement, we argued, we should place youth in the community and enroll them in evidence-based practices (EBPs) close to home.

Robust evaluations of these programs, like Multi-Systemic Therapy (MST) and Functional Family Therapy (FFT), showed that they are more cost-effective than locking up young people. Blue Prints for Healthy Youth Development and the Washington State Institute for Public Policy made the evaluations of these programs easy to understand and compare to incarceration.

Policymakers and government officials seem to have bought what we were selling: As a former staffer at a juvenile justice agency, I had to help a council member understand why every youth couldn’t and shouldn’t be in an MST program. These policymakers wanted the “gold standard” EBPs, but not every young person qualified to participate in MST.

What’s more, MST and other evidence-based programs couldn’t meet the needs of every young person in the system. Perhaps it’s our own doing that we now hear policymakers, government officials and an array of stakeholders beating the drum to implement evidence-based practices, even at the expense of other promising and innovative services, interventions and programs.

Although they can well serve youth with specific needs, evidence-based practices and programs aren’t perfect, and they aren’t the only programs that can meet a young person’s needs. They can also be very costly to implement and evaluate, demand strict fidelity to the original model and only work for the type of youth for which they were designed.

In other words, they are not meant to be tinkered with to meet the unique needs of a young person, their family, their community or the agency implementing them. This can leave out a lot of young people, perhaps the same young people who still end up in secure confinement or who have been transferred to the adult system.

The problem isn’t simply that EBPs can’t and don’t serve all youth — the problem is, in part, that we think they should. The development and promotion of these programs have been, undoubtedly, an invaluable asset for youth and the field. But our zealous advocacy for EBPs may have resulted in the devaluation of other crucial, flexible approaches to serving the large numbers of youth whose needs are not fully met by EBPs.

EBPs can have a clinical rather than individualized approach. While therapy and medications are clearly helpful, when used in isolation they can have a pathologizing effect that makes youth feel less like an asset and more like damaged goods. Treating youth like an asset is a central feature of positive youth development, an approach widely accepted, but not necessarily evidence-based.

To get around the challenges posed by the restrictiveness of EBPs, especially related to customizing services, youth-serving agencies and organizations have adopted or created programs and practices that are designed to meet the unique needs of young people in the context of their communities, ideally leveraging community resources. When implemented well, these programs are informed by the best research, practices and data about what works to help kids and are designed to meet the unique needs of the agencies that decide to use them too. They are flexible and can be adjusted quickly to meet public safety, political, social or budgetary needs.

Such programs are usually based on fundamental principles of positive youth development, like giving a young person the opportunity to learn and experience new things, develop positive relationships and a sense of belonging, and be treated like an asset. Programs like educational services, counseling, mentoring and job training, among many others, are all widely understood to provide the ingredients for success.

For cash-strapped jurisdictions, programs or policies that are not dubbed evidence-based but use evidence-based principles are a boon. Rather than use a strict, academic evaluation procedure, they seek to use a highly calibrated and robust system of oversight to make sure youth are getting high-quality services that meet their unique needs and can produce positive outcomes. These approaches can help provide the supports and services that help youth succeed in the community, making an out-of-home placement unnecessary. Here are some examples:

  • Positive Youth Justice (PYJ) is an approach that draws from positive youth development principles to meet the unique needs of justice-involved youth. PYJ includes six domains: education, work, relationships, creativity, community and health. These domains are not meant to be used in isolation, but rather as a system of supports and services that can include EBPs.
  • DC YouthLink in the District of Columbia provides supports and services to young people placed in the community along the five PYJ domains. DC YouthLink relies on a number of programs and practices with evidence-based principles, like mentoring for example. It also uses a robust system of quality assurance to ensure that young people are getting what they need to succeed, which includes going beyond recidivism measures to include positive outcomes like high school graduation, job attainment and positive attachments.
  • Wayne County, Michigan has a system of community-based programs and services that are not all evidence-based but still exposed to a system of quality checks. By building funding and service coordination of juvenile justice services into a Care Management Organization (CMO) model, the county and the nonprofit collaborators can tailor the right kind of services to the youth and their family in the context of their community and collect data to track to ensure quality service delivery.
  • The Standard Program Evaluation Protocol is a rubric designed by Mark Lipsey and his colleagues at Vanderbilt University to determine the cost-effectiveness of community-based programs. It isn’t meant to be an off-the-shelf guide to evaluation and, for now, it focuses solely on recidivism as the outcome. As a tool, it helps systems look at their whole youth population and determine the most cost-effective interventions that can be effective for youth with different risks and needs, giving the system the ability to ensure that their interventions are working without investing in a traditional evaluation.

Now that we’re a decade out from many of the larger reform efforts, it’s easier to see that evidence-based practices aren’t and shouldn’t be the only tool in the shed. Just as the field helped to advance the credibility of EBPs, so do we also need to call for other types of community-based programs rooted in the knowledge of what works to help kids.

With tens of thousands of youth still locked up — the one intervention that is clearly not evidence-based — we should have the courage, vision and ingenuity to provide customized supports and services to youth in the community rather than relying on one type of intervention.

A hammer is very effective, but only if everything is, in fact, a nail.

Amanda Petteruti is Senior Research Associate at the Justice Policy Institute and former Program Analyst for the District of Columbia’s Department of Youth Rehabilitation Services.


7 thoughts on “OP-ED: Evidence-based Practices Aren’t the Only Tool in the Shed

  1. How does one measure the compassion and understanding needed to reach these young persons. Or measure empirical impact of the socially punitive and legally bias system on this young fragile dyad. Intellectualizing or ideas about change matter not and do little to elevate real pain suffered at the hands of our legal system. Money can never buy compassion, or emotional understanding for the plight of another. The money is in punishment and prisons, very few care about little less. Families, poverty and inequality have no lobby, little money and few voices.

  2. Agree. Great article. As one of the architects of our Georgia reform, my only disappointment was the statutory requirement that all reinvestment funds captured from the costs saved must go to “evidence based programs and practices.” I now sit on the newly created reform commission and co-chair the Oversight and Implementation Committee and for me at least, it has been a bit of a frustration to convince those who control the purse string to appreciate the content of your article. I even tried to work in the distinction between “program” and “practice” and convince them to allocate monies for certain practices we do know are effective such as GPS monitoring to supplement programming for high risk youth–it was a no. But I must be fair to my colleagues who hold those purse strings and appreciate what they are struggling to achieve in our infancy of reform–instantaneous success. For us right now in Georgia, the politics remains equally essential to ensure that the reforms remain intact and that means bringing down the number of commitments with hard and fast EBPs without serious incidents and high rates of recidivism. In other words, this strict application of our law will in time be relaxed once we are settled in and our preliminary numbers are showing that it is working. You are correct–EBPs can take us only so far. Between the world of policy-making and politics there is usually some clouds with a hope of clear skies for the future. I intend to use your article when the time is right here to relax our definition of EBPs. Thank you.

  3. I think we can all agree that any movement towards a more data-driven and outcome-focused set of policies is preferable to the status quo when it comes to the juvenile justice population. Thus, interventions that may not be ‘evidenced-based’, but informed by the research and based on data would be a welcome change. A few points to consider: 1) the actual penetration of EBPs is still pretty low, Delbert Elliot of Blueprints fame estimates that fewer then 10% of youth who could benefit from EBPs are actually receiving them. EBPs may not work for everyone, but the majority of kids who it will work for are still not able to access them. 2) A key reason that policy makers have embraced EBPs is because of the rigour that it brings and the value placed on fidelity. It is because policy makers value accountability that they are drawn to EBPs. When ‘evidenced-informed’ kinds of programs are used, the stakeholders often report wide variation in the quality of the service. Such programs would benefit greatly from some form of attention to quality across sites. 3) I would dispute the notion that EBPs promote a ‘one-size-fits-all’ approach to juvenile offending. Almost by definition, an EBP will treat each family as unique and offer an individualized set of interventions. I’m actually surprised that the notion that EBPs treat every kid the same is even being promoted. The research is pretty clear that there are a variety of factors that contribute to juvenile delinquency and each child will be different. EBPs are effective in large part, because this is a concept that underlies the programs.

  4. Really appreciate this candid discussion. I too have questions about the ‘one-size-fits-all’ approach — one-size-fits few people properly. Policy makers can’t be lulled into the one-size mentality, and have to be held accountable for assuring that all the resources our kids need are available, adequately funded, adequately staffed and adequately resourced! Whether it’s the family, or the social services agencies, or our probation departments — they all ought to be adequate to the needs of all our children!
    Thanks for reassurance of my ‘hunch’!