# data[].attributes.body
Sample Associated with Video

# data[].attributes.attachment_link.description
This is "NSCC Video Part One (V7)" by Groupatwork on Vimeo, the home for high quality videos and the people who love them.

# data[].attributes.attachment_link.readable_text
We would appreciate your help in analyzing and understanding the article we have just read. To further this cause, I have added some questions that you could answer for us. You don't have to answer them all - just a minimum of two. You can answer more, if you like. But, if you can pick two that have not been answered yet then we can help each other process the important information. The questions are listed below.

# data[].attributes.attachment_link.description
We would appreciate your help in analyzing and understanding the article we have just read. To further this cause, I have added some questions that you could answer for us. You don't have to answer them all - just a minimum of two. You can answer more, if you like.

# data[].attributes.body
*Insert links for articles referenced in this video tutorial.

# data[].attributes.body
Participants can review the live results of the survey by clicking on this link:

# data[].attributes.attachment_link.description
View and analyze results from the Memorable Learning Experience SurveyMonkey survey. See actual response data, text summaries and word clouds.

# data[].attributes.body
The following video demos illustrate how a series of experiential learning activities can be used in the classroom setting.

# data[].attributes.body
This article summarizes the latest findings for the most common teaching methods used in today's classroom.

# data[].attributes.attachment_link.description
Web survey powered by SurveyMonkey.com. Create your own online survey now with SurveyMonkey's expert certified FREE templates.

# data[].attributes.body
Do you have any questions regarding the article? If so, post them in this column. If you have an answer to someone's question, please share it here.

# data[].attributes.body
What insights, learnings could you share with us based on your reading of this article?

# data[].attributes.body
The researchers describe two groups of students: "leavers" and "persisters". What impact could active learning have on those susceptible to "leaving" (giving up on their learning)?

# data[].attributes.body
The researchers state that, "If the experiments analyzed here had been conducted as randomized controlled trials of medical interventions, they may have been stopped for benefit." Question: What does this mean?

# data[].attributes.body
The researchers identified one important factor which might underestimate the impact of active learning in the studies performed to date. Question: What factor did they identify?

# data[].attributes.body
True or False: The research indicates that active learning has the highest impact on courses with 50 or fewer students.

True or False?

# data[].attributes.body
True or False: The data indicate that active learning increases student performance across STEM disciplines.

True or False?

# data[].attributes.body
Of the 29,300 students in the 67 lecturing treatments with data on failure rates how many students would have succeeded if they had been taught using active learning methods?

# data[].attributes.body
What were the average failure rates under active learning?

# data[].attributes.headline
"Traditional learning" average failure rates

# data[].attributes.subject
"Traditional learning" average failure rates

# data[].attributes.body
What were the average failure rates under traditional lecturing?

# data[].attributes.body
Question: The research indicated that students in traditional lecture courses were how many more times likely to fail than in courses with active learning?

# data[].attributes.body
What consensus definition did the researchers create for "traditional lecturing"?

# data[].attributes.body
What consensus definition did the researchers create for "active learning"?

# data[].attributes.body
Question: What were some of the learning activities that qualified as an "active learning intervention"?

# data[].attributes.body
This article documents student performance in active learning versus traditional lecturing courses. Question: How many experiments did the research team metaanalyze?

# data[].attributes.body
The article states that "constructivist versus exposition-centered methods impact student performance in undergraduate courses..." (p. 8410) Question: What is the difference between constructivist and exposition-centered teaching methods?

# data[].attributes.headline
Introduction to Active Research Questions

# data[].attributes.subject
Introduction to Active Research Questions

# data[].attributes.body
We would appreciate your help in analyzing and understanding the article we have just read. To further this cause, I have added some questions that you could answer for us. You don't have to answer them all - just a minimum of two. You can answer more, if you like. But, if you can pick two that have not been answered yet then we can help each other process the important information. The questions are listed below.

# data[].attributes.attachment_link.provider_name
Proceedings of the National Academy of Sciences

# data[].attributes.attachment_link.readable_text
Significance

The President's Council of Advisors on Science and Technology has called for a 33% increase in the number of science, technology, engineering, and mathematics (STEM) bachelor's degrees completed per year and recommended adoption of empirically validated teaching practices as critical to achieving that goal. The studies analyzed here document that active learning leads to increases in examination performance that would raise average grades by a half a letter, and that failure rates under traditional lecturing increase by 55% over the rates observed under active learning. The analysis supports theory claiming that calls to increase the number of students receiving STEM degrees could be answered, at least in part, by abandoning traditional lecturing in favor of active learning.

Abstract

To test the hypothesis that lecturing maximizes learning and course performance, we metaanalyzed 225 studies that reported data on examination scores or failure rates when comparing student performance in undergraduate science, technology, engineering, and mathematics (STEM) courses under traditional lecturing versus active learning. The effect sizes indicate that on average, student performance on examinations and concept inventories increased by 0.47 SDs under active learning ( n = 158 studies), and that the odds ratio for failing was 1.95 under traditional lecturing ( n = 67 studies). These results indicate that average examination scores improved by about 6% in active learning sections, and that students in classes with traditional lecturing were 1.5 times more likely to fail than were students in classes with active learning. Heterogeneity analyses indicated that both results hold across the STEM disciplines, that active learning increases scores on concept inventories more than on course examinations, and that active learning appears effective across all class sizes-although the greatest effects are in small ( n ≤ 50) classes. Trim and fill analyses and fail-safe n calculations suggest that the results are not due to publication bias. The results also appear robust to variation in the methodological rigor of the included studies, based on the quality of controls over student quality and instructor identity. This is the largest and most comprehensive metaanalysis of undergraduate STEM education published to date. The results raise questions about the continued use of traditional lecturing as a control in research studies, and support active learning as the preferred, empirically validated teaching practice in regular classrooms.

Results

The overall mean effect size for performance on identical or equivalent examinations, concept inventories, and other assessments was a weighted standardized mean difference of 0.47 ( Z = 9.781, P << 0.001)-meaning that on average, student performance increased by just under half a SD with active learning compared with lecturing. The overall mean effect size for failure rate was an odds ratio of 1.95 ( Z = 10.4, P << 0.001). This odds ratio is equivalent to a risk ratio of 1.5, meaning that on average, students in traditional lecture courses are 1.5 times more likely to fail than students in courses with active learning. Average failure rates were 21.8% under active learning but 33.8% under traditional lecturing-a difference that represents a 55% increase ( Fig. 1 and Fig. S1).

Heterogeneity analyses indicated no statistically significant variation among experiments based on the STEM discipline of the course in question, with respect to either examination scores ( Fig. 2A; Q = 910.537, df = 7, P = 0.160) or failure rates ( Fig. 2B; Q = 11.73, df = 6, P = 0.068). In every discipline with more than 10 experiments that met the admission criteria for the metaanalysis, average effect sizes were statistically significant for either examination scores or failure rates or both ( Fig. 2, Figs. S2 and S3, and Tables S1A and S2A). Thus, the data indicate that active learning increases student performance across the STEM disciplines.

For the data on examinations and other assessments, a heterogeneity analysis indicated that average effect sizes were lower when the outcome variable was an instructor-written course examination as opposed to performance on a concept inventory ( Fig. 3A and Table S1B; Q = 10.731, df = 1, P << 0.001). Although student achievement was higher under active learning for both types of assessments, we hypothesize that the difference in gains for examinations versus concept inventories may be due to the two types of assessments testing qualitatively different cognitive skills. This explanation is consistent with previous research indicating that active learning has a greater impact on student mastery of higher- versus lower-level cognitive skills ( 6⇓ ⇓ - 9), and the recognition that most concept inventories are designed to diagnose known misconceptions, in contrast to course examinations that emphasize content mastery or the ability to solve quantitative problems ( 10). Most concept inventories also undergo testing for validity, reliability, and readability.

Heterogeneity analyses indicated significant variation in terms of course size, with active learning having the highest impact on courses with 50 or fewer students ( Fig. 3B and Table S1C; Q = 6.726, df = 2, P = 0.035; Fig. S4). Effect sizes were statistically significant for all three categories of class size, however, indicating that active learning benefitted students in medium (51-110 students) or large (>110 students) class sizes as well.

When we metaanalyzed the data by course type and course level, we found no statistically significant difference in active learning's effect size when comparing ( i) courses for majors versus nonmajors ( Q = 0.045, df = 1, P = 0.883; Table S1D), or ( ii) introductory versus upper-division courses ( Q = 0.046, df = 1, P = 0.829; Tables S1E and S2D).

To evaluate how confident practitioners can be about these conclusions, we performed two types of analyses to assess whether the results were compromised by publication bias, i.e., the tendency for studies with low effect sizes to remain unpublished. We calculated fail-safe numbers indicating how many missing studies with an effect size of 0 would have to be published to reduce the overall effect sizes of 0.47 for examination performance and 1.95 for failure rate to preset levels that would be considered small or moderate-in this case, 0.20 and 1.1, respectively. The fail-safe numbers were high: 114 studies on examination performance and 438 studies on failure rate (SI Materials and Methods). Analyses of funnel plots ( Fig. S5) also support a lack of publication bias (SI Materials and Methods).

To assess criticisms that the literature on undergraduate STEM education is difficult to interpret because of methodological shortcomings (e.g., ref. 11), we looked for heterogeneity in effect sizes for the examination score data, based on whether experiments did or did not meet our most stringent criteria for student and instructor equivalence. We created four categories to characterize the quality of the controls over student equivalence in the active learning versus lecture treatments (SI Materials and Methods), and found that there was no heterogeneity based on methodological quality ( Q = 2.097, df = 3, P = 0.553): Experiments where students were assigned to treatments at random produced results that were indistinguishable from three types of quasirandomized designs ( Table 1). Analyzing variation with respect to controls over instructor identity also produced no evidence of heterogeneity ( Q = 0.007, df = 1, P = 0.934): More poorly controlled studies, with different instructors in the two treatment groups or with no data provided on instructor equivalence, gave equivalent results to studies with identical or randomized instructors in the two treatments ( Table 1). Thus, the overall effect size for examination data appears robust to variation in the methodological rigor of published studies.

Discussion

The data reported here indicate that active learning increases examination performance by just under half a SD and that lecturing increases failure rates by 55%. The heterogeneity analyses indicate that ( i) these increases in achievement hold across all of the STEM disciplines and occur in all class sizes, course types, and course levels; and ( ii) active learning is particularly beneficial in small classes and at increasing performance on concept inventories.

Although this is the largest and most comprehensive metaanalysis of the undergraduate STEM education literature to date, the weighted, grand mean effect size of 0.47 reported here is almost identical to the weighted, grand-mean effect sizes of 0.50 and 0.51 published in earlier metaanalyses of how alternatives to traditional lecturing impact undergraduate course performance in subsets of STEM disciplines ( 11, 12). Thus, our results are consistent with previous work by other investigators.

The grand mean effect sizes reported here are subject to important qualifications, however. For example, because struggling students are more likely to drop courses than high-achieving students, the reductions in withdrawal rates under active learning that are documented here should depress average scores on assessments-meaning that the effect size of 0.47 for examination and concept inventory scores may underestimate active learning's actual impact in the studies performed to date (SI Materials and Methods). In contrast, it is not clear whether effect sizes of this magnitude would be observed if active learning approaches were to become universal. The instructors who implemented active learning in these studies did so as volunteers. It is an open question whether student performance would increase as much if all faculty were required to implement active learning approaches.

Assuming that other instructors implement active learning and achieve the average effect size documented here, what would a shift of 0.47 SDs in examination and concept inventory scores mean to their students?

i) Students performing in the 50th percentile of a class based on traditional lecturing would, under active learning, move to the 68th percentile of that class ( 13)-meaning that instead of scoring better than 50% of the students in the class, the same individual taught with active learning would score better than 68% of the students being lectured to.

ii) According to an analysis of examination scores in three introductory STEM courses (SI Materials and Methods), a change of 0.47 SDs would produce an increase of about 6% in average examination scores and would translate to a 0.3 point increase in average final grade. On a letter-based system, medians in the courses analyzed would rise from a B− to a B or from a B to a B+.

The result for undergraduate STEM courses can also be compared with the impact of educational interventions at the precollege level. A recent review of educational interventions in the K-12 literature reports a mean effect size of 0.39 when impacts are measured with researcher-developed tests, analogous to the examination scores analyzed here, and a mean effect size of 0.24 for narrow-scope standardized tests, analogous to the concept inventories analyzed here ( 14). Thus, the effect size of active learning at the undergraduate level appears greater than the effect sizes of educational innovations in the K-12 setting, where effect sizes of 0.20 or even smaller may be considered of policy interest ( 14).

There are also at least two ways to view an odds ratio of 1.95 for the risk of failing a STEM course:

i) If the experiments analyzed here had been conducted as randomized controlled trials of medical interventions, they may have been stopped for benefit-meaning that enrolling patients in the control condition might be discontinued because the treatment being tested was clearly more beneficial. For example, a recent analysis of 143 randomized controlled medical trials that were stopped for benefit found that they had a median relative risk of 0.52, with a range of 0.22 to 0.66 ( 15). In addition, best-practice directives suggest that data management committees may allow such studies to stop for benefit if interim analyses have large sample sizes and P values under 0.001 ( 16). Both criteria were met for failure rates in the education studies we analyzed: The average relative risk was 0.64 and the P value on the overall odds ratio was << 0.001. Any analogy with biomedical trials is qualified, however, by the lack of randomized designs in studies that included data on failure rates.

ii) There were 29,300 students in the 67 lecturing treatments with data on failure rates. Given that the raw failure rate in this sample averaged 33.8% under traditional lecturing and 21.8% under active learning, the data suggest that 3,516 fewer students would have failed these STEM courses under active learning. Based on conservative assumptions (SI Materials and Methods), this translates into over US$3,500,000 in saved tuition dollars for the study population, had all students been exposed to active learning. If active learning were implemented widely, the total tuition dollars saved would be orders of magnitude larger, given that there were 21 million students enrolled in US colleges and universities alone in 2010, and that about a third of these students intended to major in STEM fields as entering freshmen ( 17, 18).

Finally, increased grades and fewer failures should make a significant impact on the pipeline problem. For example, the 2012 President's Council of Advisors on Science and Technology report calls for an additional one million STEM majors in the United States in the next decade-requiring a 33% increase from the current annual total-and notes that simply increasing the current STEM retention rate of 40% to 50% would meet three-quarters of that goal ( 5). According to a recent cohort study from the National Center for Education Statistics ( 19), there are gaps of 0.5 and 0.4 in the STEM-course grade point averages (GPAs) of first-year bachelor's and associate's degree students, respectively, who end up leaving versus persisting in STEM programs. A 0.3 "bump" in average grades with active learning would get the "leavers" close to the current performance level of "persisters." Other analyses of students who leave STEM majors indicate that increased passing rates, higher grades, and increased engagement in courses all play a positive role in retention ( 20⇓ - 22).

In addition to providing evidence that active learning can improve undergraduate STEM education, the results reported here have important implications for future research. The studies we metaanalyzed represent the first-generation of work on undergraduate STEM education, where researchers contrasted a diverse array of active learning approaches and intensities with traditional lecturing. Given our results, it is reasonable to raise concerns about the continued use of traditional lecturing as a control in future experiments. Instead, it may be more productive to focus on what we call "second-generation research": using advances in educational psychology and cognitive science to inspire changes in course design ( 23, 24), then testing hypotheses about which type of active learning is most appropriate and efficient for certain topics or student populations ( 25). Second-generation research could also explore which aspects of instructor behavior are most important for achieving the greatest gains with active learning, and elaborate on recent work indicating that underprepared and underrepresented students may benefit most from active methods. In addition, it will be important to address questions about the intensity of active learning: Is more always better? Although the time devoted to active learning was highly variable in the studies analyzed here, ranging from just 10-15% of class time being devoted to clicker questions to lecture-free "studio" environments, we were not able to evaluate the relationship between the intensity (or type) of active learning and student performance, due to lack of data (SI Materials and Methods).

As research continues, we predict that course designs inspired by second-generation studies will result in additional gains in student achievement, especially when the types of active learning interventions analyzed here-which focused solely on in-class innovations-are combined with required exercises that are completed outside of formal class sessions ( 26).

Finally, the data suggest that STEM instructors may begin to question the continued use of traditional lecturing in everyday practice, especially in light of recent work indicating that active learning confers disproportionate benefits for STEM students from disadvantaged backgrounds and for female students in male-dominated fields ( 27, 28). Although traditional lecturing has dominated undergraduate instruction for most of a millennium and continues to have strong advocates ( 29), current evidence suggests that a constructivist "ask, don't tell" approach may lead to strong increases in student performance-amplifying recent calls from policy makers and researchers to support faculty who are transforming their undergraduate STEM courses ( 5, 30).

Materials and Methods

To create a working definition of active learning, we collected written definitions from 338 audience members, before biology departmental seminars on active learning, at universities throughout the United States and Canada. We then coded elements in the responses to create the following consensus definition: Active learning engages students in the process of learning through activities and/or discussion in class, as opposed to passively listening to an expert. It emphasizes higher-order thinking and often involves group work. (See also ref. 31, p. iii).

Following Bligh ( 32), we defined traditional lecturing as "...continuous exposition by the teacher." Under this definition, student activity was assumed to be limited to taking notes and/or asking occasional and unprompted questions of the instructor.

Acknowledgments

We thank Roddy Theobald for advice on interpreting odds ratios; the many authors who provided missing data upon request (SI Materials and Methods); Colleen Craig, Daryl Pedigo, and Deborah Wiegand for supplying information on examination score standard deviations and grading thresholds; Kelly Puzio and an anonymous reviewer for advice on analyzing data from quasirandom studies; and Steven Kroiss, Carl Wieman, and William Wood for comments that improved the manuscript. M.S. was supported in part by National Science Foundation Grant 0962805.

Footnotes

Author contributions: S.F. and M.P.W. designed research; S.F., M.M., M.K.S., N.O., H.J., and M.P.W. performed research; S.F. and S.L.E. analyzed data; and S.F., S.L.E., M.M., M.K.S., N.O., H.J., and M.P.W. wrote the paper.

The authors declare no conflict of interest.

↵*This Direct Submission article had a prearranged editor.

See Commentary on page 8319.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1319030111/-/DCSupplemental.

Freely available online through the PNAS open access option.

View Abstract

# data[].attributes.attachment_link.title
Active learning increases student performance in science, engineering, and mathematics

# data[].attributes.attachment_link.description
The President's Council of Advisors on Science and Technology has called for a 33% increase in the number of science, technology, engineering, and mathematics (STEM) bachelor's degrees completed per year and recommended adoption of empirically validated teaching practices as critical to achieving that goal.

# data[].attributes.body
John Dewey: America's Scholar

# data[].attributes.attachment_link.description
Scholar A.G. Rud explains how the ideas of John Dewey-whose life spanned from the Civil War through World War II-are still very relevant to education in the 21st Century. Rud is dean of the Washington State University College of Education and co-editor of John Dewey at 150: Reflections for a new century.

# data[].attributes.body
Noam Chomsky Interview Clip