Tutoring programs are not widgets

Matthew A. Kraft Beth Schueler Grace Falken just released an important meta-analysis of 282 randomized controlled trials of tutoring (What Impacts Should We Expect from Tutoring at Scale? Exploring Meta-Analytic Generalizability). My spellcheck says that's not a word, but I trust the authors :)

Here's the most important figure, showing that as tutoring programs serve more students, their impact decreases. (In most cases, we are left to presume, the cost stays the same.)

The authors wonder whether (1) the declining effects are a product of study design, (2) as programs scale they change their program design (for the worse), (3) the declining effects are a product of tutoring at scale benefiting children with fewer needs, (4) implementation quality declines as tutoring programs scale.

I'm not ready to place my chips on one of those options. I will, however, share that the deeper I get into tutoring the more I believe that treating tutoring programs as equivalent inputs ("widgets") in an education production function is not going to yield the insights we're looking for. I think Michael Goldstein is with me.

My evidence base is not a meta-analysis, but rather the incremental improvement of a single program, Once , and the technology that enables what we do today that either didn't exist or was hopelessly expensive a decade ago.

2.5 years ago as Once was just getting started, here was the cold email I sent to principals across the country:

In the winter of 2022, when I sent that email, Once provided 15 minutes of daily, in-person, evidence-based, one-on-one, reading instruction to PreK and kindergarten students. Following high-impact tutoring best practices:

• tutoring was built into the school day
• we used routine assessments to individualize instruction
• we reflected on program-wide data to continuously improve
• we used high-quality instructional materials grounded in 50 years of research in the Science of Reading
• students were tutored daily, one-on-one
• tutors were consistently available and received ongoing oversight and coaching

And, back then, unlike any other in-person tutoring program I've encountered, every instructional session was video recorded and tutors received weekly coaching based on student data and review of those instructional sessions.

Today, 2.5 years later, our pitch hasn't changed much:

We've added a little more data around impact, but otherwise, largely the same program. We're still focused on K reading. Sessions are still 15 minutes, every day of the week. Sessions are still in-person. We still record every session and use those recordings to provide coaching to improve instruction.

Yet, 2.5 years ago we would have fallen over had we tried to deliver the program to hundreds, let alone thousands of students. I'm certain we would have seen just the declines at scale that are evident in the chart at the start of this article.

However, in the years since, at Once we've built:

• A database and dashboards to answer questions about student progress in real time in a way that was unachievable with the Google sheets formulas we used to use.
• Biweekly oral reading fluency tests in the background, invisible to students
• Codes to diagnose the causes of missed sessions
• Custom designed decodable books to accompany our curriculum
• Student comprehension support
• Sound videos and lesson demonstration videos
• Writing worksheets
• Detailed curriculum guide
• Detailed operational metrics to compare implementations, some of which were great, others of which we gradually discarded as we discovered better ones
• User surveys
• Weekly coach management and professional development meetings
• Lead classroom teacher engagement
• Motivational contests for tutors
• Implementation guide
• Flowchart for correcting student errors
• Embedded audio buttons throughout the curriculum
• Curriculum previews
• A rubric of 55 objectives for tutors that tracks progress weekly
• Automated weekly data emails to schools and districts
• Touchpoint tracking
• Scaled processes for hiring
• 3rd party bulk printing of decodables
• Custom-built portal with WebRTC video
• Professionally video-taped training
• Bachelor's degree partnership for tutors
• Monthly data decks
• Plastic fingers to avoid accidental use of touchscreen

These are just examples and there are many others not on this list coming soon. Many thanks to our in-house historian, Michael McKenna , for compiling the list of these ones over the years in preparation for our off-sites.

My point is that comparing Once today to Once 2.5 years ago would be apples to oranges even though today and 2.5 years ago we met the basic criteria for high-impact tutoring.

Similarly ten or twenty years ago you could have run our program and provided many of the same inputs and met the criteria for high-impact tutoring. With one exception, that is--video recording and review which were not, at that time, available with off-the-shelf tools in the ways we need to run Once. So ten or twenty years ago it would have been 80% the same, just with older tech, right? Wrong! Without video coaching we would have had no way to ensure consistency of instruction, and like Reading Recovery before us, our results would likely have been all over the map--some great tutors overperforming, others substantially underperforming.

Again, Once today compared to a hypothetical Once ten or twenty years ago, apples to oranges.

This school year Once (paraprofessional-led, early elementary reading, high quality instructional materials) will be serving 3,000 students (3x YoY growth!). I think Ignite Reading (virtual, early elementary reading, high quality instructional materials) is serving 10,000 or more students (although I could be off). My point is we now have examples of modern, tech-enabled tutoring programs operating at scale. What we don't have (yet) is RCT's "proving" (I use quotes because educational results can be difficult to replicate--that's why we need lots of studies) the precise impact.

If you work in philanthropy and you have capital available to help us run RCTs, please get in touch. By proving what modern tutoring programs are starting to see anecdotally, we'll unlock the budgets necessary to serve huge numbers of students, repair some of the post-Covid learning loss damage, put points on the board for NAEP, and, in the end, create a lasting change in education.

But if you're looking for effects at scale from tutoring in general as an intervention, I think Matthew A. Kraft et al see the writing on the wall.