Social innovation, effectual reasoning, and developmental evaluation.
I am so grateful to have been a part of an initiative of Pittsburgh Action Against Rape to disseminate sexual violence prevention messages among male athletes. I got to study the evidence-based violence prevention program Coaching Boys Into Men, and grapple with ideas about about social innovation, effectual reasoning, and developmental evaluation with prevention champion Julie Evans and her team.
Coaching Boys Into Men as Social Innovation
Coaching Boys Into Men (CBIM) is an evidence-based prevention program for male coaches and athletes aimed at stopping sexual and relationship violence before it happens. Throughout the season, coaches facilitate a series of conversations with their players about: respect, personal responsibility, insulting language, disrespectful behavior towards women and girls, digital disrespect, consent, bragging about sexual reputation, aggression on and off the field, pressure & intimidation in relationships, setting boundaries in relationships, modeling respect & promoting equality.
CBIM, like most sexual violence prevention efforts, qualifies as social innovation because it is a more effective, efficient, sustainable, and just solution to the social problem of sexual violence than many existing criminal justice responses to sexual violence perpetration.
Here's evidence on CBIM's effectiveness, documented by Dr. Liz Miller and colleagues. Because CBIM had already been field tested, PAAR's evaluation questions did not focus on program's effectiveness. Rather, PAAR's team needed answers to questions such as these scratched across this whiteboard in the South Side Flats. The goal was to recruit high school coaches, secure the coaches’ agreement to implement CBIM, and support those coaches in carrying out CBIM with their teams.
We designed an implementation study around the question — How do you find and support change agents? — to evaluate the prevention support PAAR was providing to coaches. Below is the one-pager of our conclusions.
The Final Step of the Public Health Model Requires a Shift in Thinking
Whereas the first three steps of the public health approach to violence prevention — define the problem, identify risk and protective factors, and develop and test prevention strategies — often rely on controlled research designs and causal reasoning, the last step — assure widespread adoption — requires a shift in thinking. The primary task is “selling” new ideas, beliefs, behaviors, or social policies to targeted groups.
Revisiting the diffusion of innovations theory was helpful in understanding coaches of male student-athletes as early adopters (EAs). EAs watch and follow innovators on the cutting edge. They, too, are visionaries and risk takers so the unknown appeals to them and they are continuously looking for new opportunities. Because EAs tend to be leaders and role models in their social settings, they can provide the “stamp of approval” that reduces risk for the early majority and helps spread acceptance of the innovation.
We also got to apply some ideas from the society for effectual action.
As shown in the figure at the top of this post, effectual reasoning stands in contrast to causal thinking. As I listened to PAAR's prevention team, which included male high school coaches, talk about their work, it struck me that they were thinking more like entrepreneurs than educators or program managers. In the early stages of this work, I asked, “how are you training your early-adopting coaches?”
I learned that, in addition to holding classroom-style training, PAAR coaches were also meeting one-on-one in public spaces like parking lots, basketball courts, and training rooms to do “on-the-fly” training with other coaches they wanted to adopt the program. It was an important selling point that the CBIM program could be worked right into what coaches were already doing, without a lot of training. Requiring interested coaches to come to a day-long training workshop, or even half-day workshop, felt tricky.
Evaluating PAAR’s efforts to take CBIM to scale required a new way of thinking about program development and evaluation. The team at PAAR was willing to embrace a new idea; and so we worked together to apply the five principles of Professor Sarasvathy’s concept of effectuation, what I refer to as effectual thinking, to our CBIM evaluation project.
The Work Unfolded as a Series of Implementation Trials
Over the course of two years, the PAAR coaches applied Bird-in-Hand to leverage existing relationships in order to implement multiple rounds of CBIM, which are described on page 7 of the report shown below. For example, the first implementation trial made use of an existing partnership that PAAR had implementing prevention programming in a junior high school setting. After each implementation trial, the evaluation team convened in order to facilitate a learning and goal-setting process to guide the next implementation trial.
Social Innovations Can Be Challenging to Evaluate
Complex real-world conditions in communities demand our patience and persistence even when we are evaluating the most “tried-and-true” programs. When the program is unknown (as CBIM was), questions conventional beliefs about gender and sex and masculinity (which are deeply personal topics), attempts to alter high school football (which is sacred in many communities), and is implemented by coaches who are not getting paid to do the program, it becomes that-much-more challenging.
Developmental Evaluation Acknowledges the Complexity of Social Innovation
In designing evaluation activities that were consistent with the effectual reasoning leading PAAR’s CBIM implementation work, we incorporated some of Scriven’s (1991) early ideas about goal-free evaluation (GBE), and leaned on Patton’s (2011) more recent guidance on developmental evaluation (DE).
Page 8 of the report summarizes how both innovation and complexity are accounted for in the emerging practice of DE.
Developmental Evaluation Is Centered on the Innovators' Commitment to Make a Difference
For PAAR's CBIM Implementation Project, developmental evaluation helped us: 1) come to terms with the messy space we were working in, and 2) reframe expectations around how to document value, significance, and worth of the work. We used two metaphors when discussing this project, which are summarized on page 9 of the report.
While the developmental evaluation (DE) approach is "not the solution to every situation," we successfully applied DE principles to gather and process information that created insight; make decisions about new program elements, e.g., PAAR's Coach-the-Coach prevention support; and ultimately, document the value and worthiness of PAAR's community prevention work.
Page 10 of the report, shown below, summarized three of the most significant and valuable change levers in PAAR’s efforts, followed another 49 pages of evaluation data that provided evidence of the hallmark successes of the project.
It was my absolute pleasure to join Southwest PA says NO MORE, the FISA Foundation, and PAAR to work on providing men and boys some tools for preventing sexual violence and modeling respect and equality.
To talk to me about how you might apply effectual thinking in your work; or to share your experiences with DE; please comment below or send me an email. I’d love to hear from you!