Last month, I finally gave up on a much loved, but completely worn-out pair of sneakers and started shopping online for a replacement. A week later, despite reading reviews for at least 15 different shoes and carefully studying a number of sizing charts, I was the proud owner of two poorly-fitting and incredibly uncomfortable pairs of shoes. I had done my research but, in the end, there simply wasn’t a substitute for getting to actually try on the shoes before buying them. But this is not a post about sneakers (obviously). This is a blog about social-psychological interventions.
Researchers often run into a similar pitfall as they look to adapt their interventions to the people, or context, that they are trying to benefit. In the same way that knowing your shoe size doesn’t necessarily mean that you’ll buy the perfect shoe, adapting your intervention exclusively based on a priori information about your context – whether that’s your classroom, your company, or even your family – doesn’t mean that your work will be well-tailored to the people and situations that you’re working with. The best way to tailor your intervention to your context is by evaluating how it fits once it is administered. The Motivate Lab dealt with this issue firsthand in our four-year ongoing project designed to increase pass rates in introductory math courses at Valencia College, a community college system in Florida.
Forty percent of Valencia students in these courses do not pass them, and instructors point to students not finding the course material relevant as a key barrier to success. This project, funded by the National Science Foundation, looks to help solve this problem through utility value interventions, which prompt students to find personal relevance in their coursework. The intervention is administered as brief online activities assigned throughout the semester.
How can you use math to handle issues related to a drought in California?
One of these activities prompts students to see how math can be used to address real world issues by presenting examples about how math can help solve issues related to a drought in California. In the process of evaluating how our intervention fit our context, we realized that students were often not finding any personal relevance in the activity. While we did adapt the drought activity a bit based on prior feedback from Valencia instructors, it turns out that these students were having trouble relating to droughts that occur all the way across the country. Students’ quantitative ratings of how helpful this activity was in getting them to connect math to real life revealed that, on average, students did not find this activity as helpful as we hoped. Similarly, the qualitative data, short essays that students wrote about why the activity was (not) helpful showed that students often cited never having dealt with problems related to droughts. If you’ve ever been to Florida, you’re probably not surprised. Using the quantitative data in conjunction with our qualitative data allowed us to 1. empirically establish that the activity was not particularly helpful for students and 2. Rethink our intervention design process in order to create a more relevant version of the activity.
How can you use math to handle issues related to hurricane season?
Based on feedback from our partners at the community college and participant responses to a number of short-answer essays, we began to write a new activity based around how math could be used to solve a real world problem that many of these students in Florida had dealt with first-hand—hurricane season. Furthermore, we learned from focus groups conducted with students at the community college that they found math most relevant when it was connected to money. After workshopping this new activity with other education professionals and faculty from the community college and piloting a preliminary version of the activity in the fall of 2017, we finalized an activity featuring two quotes about how math can be used to deal with financial concerns that students said they faced during hurricane season (buying insurance, budgeting for repairs, etc.).
In an effort to make sure this redesigned activity was, indeed, more useful to Valencia students, students were randomly selected to respond to either the original drought or the revised hurricane activity. Preliminary results indicate that students in the hurricane activity group rated it as significantly more helpful in showing them how math connects to real life (see graph below) and that students who found the activity “Helpful” or “Very helpful” were 5% more likely to pass their math course.
Student ratings of how helpful they found the activity, either the original drought one or the revised hurricane one, in getting them to connect math to real life indicate that students who saw the hurricane activity were less likely to rate it as not being helpful and more likely to rate it as being “Helpful” or “Very helpful”.
How can you customize for your context?
Assuming that I could order the perfect pair of shoes online simply because I knew my shoe size and read a couple reviews, ultimately left me with two pairs of poorly-fitting sneakers. Similarly, making the assumption that the a priori information you have about your context will ensure that your work is tailored to your context may result in an intervention that doesn’t work the way you expect. Just like there’s no replacement for trying on the shoe; it’s impossible to determine whether your intervention fits its context without soliciting direct quantitative and qualitative feedback from the people who are supposed to benefit and using it to iterate on your work. This lesson can be applied outside of research. Constantly evaluating and revising your work based on feedback from the people you work with about their characteristics, challenges, and concerns allows you to best customize your work, no matter the field, to your context.
David is a research specialist at the Motivate Lab. When he’s not geeking out over interventions to remove barriers to motivation in community college math, you can find him hiking all over Central Virginia, actively avoiding the UVA Corner, or hanging out with his four-legged best friend, Rasshi.