The headline on Steve Lohr’s Bits post sounds pretty definitive: “Study Finds That Online Education Beats the Classroom.” And the quote that Lohr gets from the study’s lead author, Barbara Means, sounds equally definitive: “The study’s major significance lies in demonstrating that online learning today is not just better than nothing — it actually tends to be better than conventional instruction.”

Predictably, Lohr’s post is now inspiring even more extreme summaries of the study. “Want to learn better?” screams the headline at the misnamed SmartPlanet.com. “Crack open your laptop (and ditch the classroom).” Chimes in Podcasting News: “Why Go to School? Study Finds Students Do Better Online.”

But the study itself, which was conducted by SRI International for the US Department of Education, is considerably less definitive than the coverage would have you believe. Before school boards start firing teachers and shuttering classrooms, they might want to read the actual report.

The SRI study is a “meta-analysis,” or a “study of studies.” The authors reviewed research, dating back to 1996, of online education. Of the 1,132 studies they found, they focused their analysis on just 51 “experimental and quasi-experimental studies” that provided sufficient data to directly compare the “learning outcomes” of online courses with those of traditional “face-to-face” courses. In many cases, however, the “online courses” also involved face-to-face instruction; in other words, the online instruction supplemented (rather than replaced) classroom instruction. Only about half of the 51 analyzed studies involved purely online courses. Also, 19 of the 51 studies involved courses that lasted less than a month. And, as the authors note, “many of the studies suffered from weaknesses such as small sample sizes; failure to report retention rates for students in the conditions being contrasted; and, in many cases, potential bias stemming from the authors’ dual roles as experimenters and instructors.”

With those limitations in mind, the authors’ overall conclusion is as follows: “The corpus of 51 [experiments] was sufficient to demonstrate that in recent applications, online learning has been modestly more effective, on average, than the traditional face-to-face instruction with which it has been compared.”

But the authors also provide some important additional caveats:

First, while the intent of this meta-analysis is “to provide policy-makers, administrators and educators with research-based guidance about how to implement online learning for K–12 education and teacher preparation,” the study doesn’t actually look at K-12 education. In fact, of the 51 studies analyzed, only five were drawn from K-12 education – and none of those involved purely online instruction. The bulk of the studies involved instruction in college or graduate school or in professional training courses, particularly instruction in “medicine or health care” but also in “computer science, teacher education, social science, mathematics, languages, science and business.” In fact, when the five K-12 studies were examined in isolation, they revealed no statistically meaningful benefit from online learning (see page xvii of the report). As the authors emphasize, it would be rash to conclude that the study results will apply to K-12 schools.

Second, the finding that online or online-supplemented courses have slightly better learning outcomes may not have anything to do with the fact that they were online. It may simply be that they involved more instructional time than the strictly classroom courses. This is a point that the authors make repeatedly in their report. For example: “Despite what appears to be strong support for online learning applications, the studies in this meta-analysis do not demonstrate that online learning is superior as a medium. In many of the studies showing an advantage for online learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages.” The findings of the meta-analysis, the authors later reiterate, “should not be construed as demonstrating that online learning is superior as a medium.”

Finally, it should be noted that, of the 51 experiments studied, only 11 actually showed a statistically significant advantage to online instruction. (The authors don’t specify how many of those 11 involved purely online instruction versus classroom/online hybrids. Nor do they specify how many involved vocational instruction.) Two of the experiments showed a statistically significant advantage to face-to-face instruction. The vast majority of the studies showed no meaningful differences in outcomes.

Sometimes, the caveats in a study speak louder than the findings. I think that’s the case here. A suggestion that classroom instruction supplemented by online exercises can lead to more learning than classroom instruction alone would hardly come as a surprise. But that’s about the only firm conclusion I draw from this meta-analysis.