Improving improvements

Lots of ideas work fine in theory but not so much in the real world.

Ask education researchers Thomas Kane, Ph.D. ’91, Jennifer Ash, Dave Hersh, Chris Avery and their teams, who’ve come to this wisdom after years of mixed results in finding concrete ways to improve education in American schools.

“Progress will come from surprises,” said Kane, faculty director of Harvard’s Center for Education Policy Research (CEPR). He recalled one instance when “there was a study of textbooks where we were really confident — we’re going to find these big differences for people using more or less effective textbooks — and there was just no difference.”

So in 2015 Kane and a group of other education researchers at Harvard decided to create Proving Ground, a project that brings tools instead of hypotheses to school districts around the country, including three urban school districts in Ohio. The key is that instead of prescribing solutions, the team helps districts analyze their own data to decide which issues might be most beneficial to address.

“Rather than trying to get them to pay attention to some study, answering somebody else’s question, we’re trying to make it easier for them to answer their own questions, because those will be the questions that they’ll act on,” explained Kane, who has also worked for the Bill and Melinda Gates Foundation and as an economist in the Clinton administration.

After Proving Ground worked with its first batch of schools, the team found that one of the biggest issues all the districts wanted to address was absenteeism — students who regularly miss or skip school run a greater risk of falling behind and eventually dropping out.

The solutions that proved to be the most effective turned out to be some of the simplest.

“The two examples that all of our Ohio partners have tried out were elementary postcards and periodic personalized robocalls,” explained Hersh, the director of Proving Ground. These low-cost interventions consisted of either automatically calling a parent or guardian with a student’s attendance record, or a more personal postcard, written by the teacher, explaining what that student missed on the days they were absent. “We had pretty solid effects there. We generally saw anywhere from a 3 to 6 percent reduction in absenteeism.”

After the success of this approach in urban schools, CEPR was looking for opportunities to try it in rural schools. A federal grant made it possible for Harvard to create the National Center for Rural Education Research Networks (NCRERN) to help rural districts in Ohio and Upstate New York make use of the Proving Ground approach.

“A big barrier to doing research with rural schools is that you simply don’t have the numbers to do an evaluation in an individual district,” said Ash, director of NCRERN, referring to that critical mass of data needed to be certain that a trend is real and not just an aberration.

Ash, who brings years of experience working in and with rural schools, had her team pool data from a sample of rural schools in Ohio and Upstate New York that was large enough to be helpful in finding common issues they all faced.

And then COVID-19 hit. Ash and her team adapted quickly, bringing an in-person conference online and brainstorming ways to make solutions work in this new reality.

“We’re still here for you,” Ash told the schools. “We will figure out a way to adapt every intervention that was originally going to be in-person to be able to be done online.”

And Ash and her team at NCRERN have done just that. In addition to creating modified versions of the absenteeism remedies, they have also created COVID-specific solutions around family engagement.

“‘We need to do a better job communicating with our families,’” Ash heard from the schools. “‘They don’t know all of the resources we offer. We have a school-based health clinic; we have a winter coat closet. They don’t know about it and they don’t have a strong relationship with us.’” So Ash and her team helped create ways to increase two-way communication between the parents and the schools.

Chris Avery, who focuses on public policy and college preparedness at Proving Ground, said that while the pandemic has been disruptive, educators also gained experience that may lead to improvements.

“Some of the things are going to be slapdash. But some of the things may really work. And as Tom has described, we might be surprised at some of the things that work.” Avery explained that innovations in distance learning, for instance, could prove especially helpful for rural schools.

Reflecting on why this work is important, Hersh said, “Education is the single highest leverage place you can devote your energy to if you’re trying to improve outcomes for the folks who need those improvements the most. The highest-leverage way you can improve education is by institutionalizing, in education agencies, an ability to get better.”

Kane added, “Rather than any single idea, the thing that will make a really big difference in education is a system for testing ideas.”

This story is part of the To Serve Better series, exploring connections between Harvard and neighborhoods across the United States.