# data[].attributes.body
Used with the Google Expeditions app in the classroom.

# data[].attributes.attachment_link.title
Expeditions: Take your students to places a school bus can't

# data[].attributes.body
GeoGebra app for 3D graphing.

# data[].attributes.body
Example of augmented reality in geometry

# data[].attributes.body
A spray used to coat surface to make it a "touch screen".

# data[].attributes.attachment_link.title
Electrick: Low-Cost Touch Sensing Using Electric Field Tomography

# data[].attributes.body
New technological advances in touch screens.

# data[].attributes.attachment_link.title
PaperTab: Revolutionary paper tablet reveals future tablets to be thin and flexible as paper.

# data[].attributes.body
Explanation of SAMR Model. The levels of using technology in the classroom.

# data[].attributes.body
Supports the notion that students involved in learning achieve more.

# data[].attributes.attachment_link.readable_text
Significance

The President's Council of Advisors on Science and Technology has called for a 33% increase in the number of science, technology, engineering, and mathematics (STEM) bachelor's degrees completed per year and recommended adoption of empirically validated teaching practices as critical to achieving that goal. The studies analyzed here document that active learning leads to increases in examination performance that would raise average grades by a half a letter, and that failure rates under traditional lecturing increase by 55% over the rates observed under active learning. The analysis supports theory claiming that calls to increase the number of students receiving STEM degrees could be answered, at least in part, by abandoning traditional lecturing in favor of active learning.

Abstract

To test the hypothesis that lecturing maximizes learning and course performance, we metaanalyzed 225 studies that reported data on examination scores or failure rates when comparing student performance in undergraduate science, technology, engineering, and mathematics (STEM) courses under traditional lecturing versus active learning. The effect sizes indicate that on average, student performance on examinations and concept inventories increased by 0.47 SDs under active learning ( n = 158 studies), and that the odds ratio for failing was 1.95 under traditional lecturing ( n = 67 studies). These results indicate that average examination scores improved by about 6% in active learning sections, and that students in classes with traditional lecturing were 1.5 times more likely to fail than were students in classes with active learning. Heterogeneity analyses indicated that both results hold across the STEM disciplines, that active learning increases scores on concept inventories more than on course examinations, and that active learning appears effective across all class sizes-although the greatest effects are in small ( n ≤ 50) classes. Trim and fill analyses and fail-safe n calculations suggest that the results are not due to publication bias. The results also appear robust to variation in the methodological rigor of the included studies, based on the quality of controls over student quality and instructor identity. This is the largest and most comprehensive metaanalysis of undergraduate STEM education published to date. The results raise questions about the continued use of traditional lecturing as a control in research studies, and support active learning as the preferred, empirically validated teaching practice in regular classrooms.

Results

The overall mean effect size for performance on identical or equivalent examinations, concept inventories, and other assessments was a weighted standardized mean difference of 0.47 ( Z = 9.781, P << 0.001)-meaning that on average, student performance increased by just under half a SD with active learning compared with lecturing. The overall mean effect size for failure rate was an odds ratio of 1.95 ( Z = 10.4, P << 0.001). This odds ratio is equivalent to a risk ratio of 1.5, meaning that on average, students in traditional lecture courses are 1.5 times more likely to fail than students in courses with active learning. Average failure rates were 21.8% under active learning but 33.8% under traditional lecturing-a difference that represents a 55% increase ( Fig. 1 and Fig. S1).

Heterogeneity analyses indicated no statistically significant variation among experiments based on the STEM discipline of the course in question, with respect to either examination scores ( Fig. 2A; Q = 910.537, df = 7, P = 0.160) or failure rates ( Fig. 2B; Q = 11.73, df = 6, P = 0.068). In every discipline with more than 10 experiments that met the admission criteria for the metaanalysis, average effect sizes were statistically significant for either examination scores or failure rates or both ( Fig. 2, Figs. S2 and S3, and Tables S1A and S2A). Thus, the data indicate that active learning increases student performance across the STEM disciplines.

For the data on examinations and other assessments, a heterogeneity analysis indicated that average effect sizes were lower when the outcome variable was an instructor-written course examination as opposed to performance on a concept inventory ( Fig. 3A and Table S1B; Q = 10.731, df = 1, P << 0.001). Although student achievement was higher under active learning for both types of assessments, we hypothesize that the difference in gains for examinations versus concept inventories may be due to the two types of assessments testing qualitatively different cognitive skills. This explanation is consistent with previous research indicating that active learning has a greater impact on student mastery of higher- versus lower-level cognitive skills ( 6⇓ ⇓ - 9), and the recognition that most concept inventories are designed to diagnose known misconceptions, in contrast to course examinations that emphasize content mastery or the ability to solve quantitative problems ( 10). Most concept inventories also undergo testing for validity, reliability, and readability.

Heterogeneity analyses indicated significant variation in terms of course size, with active learning having the highest impact on courses with 50 or fewer students ( Fig. 3B and Table S1C; Q = 6.726, df = 2, P = 0.035; Fig. S4). Effect sizes were statistically significant for all three categories of class size, however, indicating that active learning benefitted students in medium (51-110 students) or large (>110 students) class sizes as well.

When we metaanalyzed the data by course type and course level, we found no statistically significant difference in active learning's effect size when comparing ( i) courses for majors versus nonmajors ( Q = 0.045, df = 1, P = 0.883; Table S1D), or ( ii) introductory versus upper-division courses ( Q = 0.046, df = 1, P = 0.829; Tables S1E and S2D).

To evaluate how confident practitioners can be about these conclusions, we performed two types of analyses to assess whether the results were compromised by publication bias, i.e., the tendency for studies with low effect sizes to remain unpublished. We calculated fail-safe numbers indicating how many missing studies with an effect size of 0 would have to be published to reduce the overall effect sizes of 0.47 for examination performance and 1.95 for failure rate to preset levels that would be considered small or moderate-in this case, 0.20 and 1.1, respectively. The fail-safe numbers were high: 114 studies on examination performance and 438 studies on failure rate (SI Materials and Methods). Analyses of funnel plots ( Fig. S5) also support a lack of publication bias (SI Materials and Methods).

To assess criticisms that the literature on undergraduate STEM education is difficult to interpret because of methodological shortcomings (e.g., ref. 11), we looked for heterogeneity in effect sizes for the examination score data, based on whether experiments did or did not meet our most stringent criteria for student and instructor equivalence. We created four categories to characterize the quality of the controls over student equivalence in the active learning versus lecture treatments (SI Materials and Methods), and found that there was no heterogeneity based on methodological quality ( Q = 2.097, df = 3, P = 0.553): Experiments where students were assigned to treatments at random produced results that were indistinguishable from three types of quasirandomized designs ( Table 1). Analyzing variation with respect to controls over instructor identity also produced no evidence of heterogeneity ( Q = 0.007, df = 1, P = 0.934): More poorly controlled studies, with different instructors in the two treatment groups or with no data provided on instructor equivalence, gave equivalent results to studies with identical or randomized instructors in the two treatments ( Table 1). Thus, the overall effect size for examination data appears robust to variation in the methodological rigor of published studies.

Discussion

The data reported here indicate that active learning increases examination performance by just under half a SD and that lecturing increases failure rates by 55%. The heterogeneity analyses indicate that ( i) these increases in achievement hold across all of the STEM disciplines and occur in all class sizes, course types, and course levels; and ( ii) active learning is particularly beneficial in small classes and at increasing performance on concept inventories.

Although this is the largest and most comprehensive metaanalysis of the undergraduate STEM education literature to date, the weighted, grand mean effect size of 0.47 reported here is almost identical to the weighted, grand-mean effect sizes of 0.50 and 0.51 published in earlier metaanalyses of how alternatives to traditional lecturing impact undergraduate course performance in subsets of STEM disciplines ( 11, 12). Thus, our results are consistent with previous work by other investigators.

The grand mean effect sizes reported here are subject to important qualifications, however. For example, because struggling students are more likely to drop courses than high-achieving students, the reductions in withdrawal rates under active learning that are documented here should depress average scores on assessments-meaning that the effect size of 0.47 for examination and concept inventory scores may underestimate active learning's actual impact in the studies performed to date (SI Materials and Methods). In contrast, it is not clear whether effect sizes of this magnitude would be observed if active learning approaches were to become universal. The instructors who implemented active learning in these studies did so as volunteers. It is an open question whether student performance would increase as much if all faculty were required to implement active learning approaches.

Assuming that other instructors implement active learning and achieve the average effect size documented here, what would a shift of 0.47 SDs in examination and concept inventory scores mean to their students?

i) Students performing in the 50th percentile of a class based on traditional lecturing would, under active learning, move to the 68th percentile of that class ( 13)-meaning that instead of scoring better than 50% of the students in the class, the same individual taught with active learning would score better than 68% of the students being lectured to.

ii) According to an analysis of examination scores in three introductory STEM courses (SI Materials and Methods), a change of 0.47 SDs would produce an increase of about 6% in average examination scores and would translate to a 0.3 point increase in average final grade. On a letter-based system, medians in the courses analyzed would rise from a B− to a B or from a B to a B+.

The result for undergraduate STEM courses can also be compared with the impact of educational interventions at the precollege level. A recent review of educational interventions in the K-12 literature reports a mean effect size of 0.39 when impacts are measured with researcher-developed tests, analogous to the examination scores analyzed here, and a mean effect size of 0.24 for narrow-scope standardized tests, analogous to the concept inventories analyzed here ( 14). Thus, the effect size of active learning at the undergraduate level appears greater than the effect sizes of educational innovations in the K-12 setting, where effect sizes of 0.20 or even smaller may be considered of policy interest ( 14).

There are also at least two ways to view an odds ratio of 1.95 for the risk of failing a STEM course:

i) If the experiments analyzed here had been conducted as randomized controlled trials of medical interventions, they may have been stopped for benefit-meaning that enrolling patients in the control condition might be discontinued because the treatment being tested was clearly more beneficial. For example, a recent analysis of 143 randomized controlled medical trials that were stopped for benefit found that they had a median relative risk of 0.52, with a range of 0.22 to 0.66 ( 15). In addition, best-practice directives suggest that data management committees may allow such studies to stop for benefit if interim analyses have large sample sizes and P values under 0.001 ( 16). Both criteria were met for failure rates in the education studies we analyzed: The average relative risk was 0.64 and the P value on the overall odds ratio was << 0.001. Any analogy with biomedical trials is qualified, however, by the lack of randomized designs in studies that included data on failure rates.

ii) There were 29,300 students in the 67 lecturing treatments with data on failure rates. Given that the raw failure rate in this sample averaged 33.8% under traditional lecturing and 21.8% under active learning, the data suggest that 3,516 fewer students would have failed these STEM courses under active learning. Based on conservative assumptions (SI Materials and Methods), this translates into over US$3,500,000 in saved tuition dollars for the study population, had all students been exposed to active learning. If active learning were implemented widely, the total tuition dollars saved would be orders of magnitude larger, given that there were 21 million students enrolled in US colleges and universities alone in 2010, and that about a third of these students intended to major in STEM fields as entering freshmen ( 17, 18).

Finally, increased grades and fewer failures should make a significant impact on the pipeline problem. For example, the 2012 President's Council of Advisors on Science and Technology report calls for an additional one million STEM majors in the United States in the next decade-requiring a 33% increase from the current annual total-and notes that simply increasing the current STEM retention rate of 40% to 50% would meet three-quarters of that goal ( 5). According to a recent cohort study from the National Center for Education Statistics ( 19), there are gaps of 0.5 and 0.4 in the STEM-course grade point averages (GPAs) of first-year bachelor's and associate's degree students, respectively, who end up leaving versus persisting in STEM programs. A 0.3 "bump" in average grades with active learning would get the "leavers" close to the current performance level of "persisters." Other analyses of students who leave STEM majors indicate that increased passing rates, higher grades, and increased engagement in courses all play a positive role in retention ( 20⇓ - 22).

In addition to providing evidence that active learning can improve undergraduate STEM education, the results reported here have important implications for future research. The studies we metaanalyzed represent the first-generation of work on undergraduate STEM education, where researchers contrasted a diverse array of active learning approaches and intensities with traditional lecturing. Given our results, it is reasonable to raise concerns about the continued use of traditional lecturing as a control in future experiments. Instead, it may be more productive to focus on what we call "second-generation research": using advances in educational psychology and cognitive science to inspire changes in course design ( 23, 24), then testing hypotheses about which type of active learning is most appropriate and efficient for certain topics or student populations ( 25). Second-generation research could also explore which aspects of instructor behavior are most important for achieving the greatest gains with active learning, and elaborate on recent work indicating that underprepared and underrepresented students may benefit most from active methods. In addition, it will be important to address questions about the intensity of active learning: Is more always better? Although the time devoted to active learning was highly variable in the studies analyzed here, ranging from just 10-15% of class time being devoted to clicker questions to lecture-free "studio" environments, we were not able to evaluate the relationship between the intensity (or type) of active learning and student performance, due to lack of data (SI Materials and Methods).

As research continues, we predict that course designs inspired by second-generation studies will result in additional gains in student achievement, especially when the types of active learning interventions analyzed here-which focused solely on in-class innovations-are combined with required exercises that are completed outside of formal class sessions ( 26).

Finally, the data suggest that STEM instructors may begin to question the continued use of traditional lecturing in everyday practice, especially in light of recent work indicating that active learning confers disproportionate benefits for STEM students from disadvantaged backgrounds and for female students in male-dominated fields ( 27, 28). Although traditional lecturing has dominated undergraduate instruction for most of a millennium and continues to have strong advocates ( 29), current evidence suggests that a constructivist "ask, don't tell" approach may lead to strong increases in student performance-amplifying recent calls from policy makers and researchers to support faculty who are transforming their undergraduate STEM courses ( 5, 30).

Materials and Methods

To create a working definition of active learning, we collected written definitions from 338 audience members, before biology departmental seminars on active learning, at universities throughout the United States and Canada. We then coded elements in the responses to create the following consensus definition: Active learning engages students in the process of learning through activities and/or discussion in class, as opposed to passively listening to an expert. It emphasizes higher-order thinking and often involves group work. (See also ref. 31, p. iii).

Following Bligh ( 32), we defined traditional lecturing as "...continuous exposition by the teacher." Under this definition, student activity was assumed to be limited to taking notes and/or asking occasional and unprompted questions of the instructor.

Acknowledgments

We thank Roddy Theobald for advice on interpreting odds ratios; the many authors who provided missing data upon request (SI Materials and Methods); Colleen Craig, Daryl Pedigo, and Deborah Wiegand for supplying information on examination score standard deviations and grading thresholds; Kelly Puzio and an anonymous reviewer for advice on analyzing data from quasirandom studies; and Steven Kroiss, Carl Wieman, and William Wood for comments that improved the manuscript. M.S. was supported in part by National Science Foundation Grant 0962805.

Footnotes

Author contributions: S.F. and M.P.W. designed research; S.F., M.M., M.K.S., N.O., H.J., and M.P.W. performed research; S.F. and S.L.E. analyzed data; and S.F., S.L.E., M.M., M.K.S., N.O., H.J., and M.P.W. wrote the paper.

The authors declare no conflict of interest.

↵*This Direct Submission article had a prearranged editor.

See Commentary on page 8319.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1319030111/-/DCSupplemental.

Freely available online through the PNAS open access option.

View Abstract

# data[].attributes.attachment_link.title
Active learning increases student performance in science, engineering, and mathematics

# data[].attributes.attachment_link.description
The President's Council of Advisors on Science and Technology has called for a 33% increase in the number of science, technology, engineering, and mathematics (STEM) bachelor's degrees completed per year and recommended adoption of empirically validated teaching practices as critical to achieving that goal.

# data[].attributes.attachment_link.readable_text
2. Digital Commerce is the electronic buying and selling of goods and focuses on the tools and safeguards in place to assist those buying, selling, banking, or using money in any way in the digital space. Career and technical education use the tools of technology to show students the path for their future.

3. Digital Communication and Collaboration is the electronic exchange of information. All users need to define how they will share their thoughts so that others understand the message. For students struggling to understand their place in the world, technology can help them find their own voices and express themselves.

4. Digital Etiquette refers to electronic standards of conduct or procedures and has to do with the process of thinking about others when using digital devices. Teachers can include Digital Etiquette as part of the classroom rules or academic goals. Whether in the classroom or online, being aware of others is an important idea for everyone.

5. Digital Fluency is the process of understanding technology and its use. The better educated or "digitally fluent," students are, the more likely they are to make good decisions online, like supporting others instead of making negative comments. Digital literacy includes the discussion of media literacy and the ability to discern good information from poor, such as "fake news" from real news.

6. Digital Health and Welfare refers to the physical and psychological well-being in a digital world. Technology provides many opportunities and enjoyment, but knowing how to segment use with the needs of ourselves and others is key to a healthy, balanced life. Educators, especially in 1:1 schools or classrooms need to ask the question of how much screen time is appropriate for students. Common Sense media has developed a guide on this topic.

7. Digital Law refers to the electronic responsibility for actions and deeds and has to do with the creation of rules and policy that address issues related to the online world. Just as in the real world, the online world has had to create structure to protect those using these digital devices from harm. Support for issues such as cyberbullying and sexting are available from School Resource Officers and other school counsel. Administrators need to come up with positive approaches to these issues in their schools and districts.

8. Digital Rights and Responsibility are those requirements and freedoms extended to everyone in a digital world. This area of Digital Citizenship is about helping students understand that when they are provided opportunities, such as the access to the Internet and use of online products, they need to be diligent in helping others as well, such as informing adults of potential problems. Educators must help students understand that protecting others both online and in the real world are essential skills to have.

9. Digital Security and Privacy is the electronic precautions to guarantee safety. Viruses, worms and other bots can be passed along from one system to another just like an illness. When using devices in school or at home, understanding and being aware of attacks and how to prevent them are important skills for today and into the future.

Check out Amanda Brace's site - Ed Tech Endeavours - 9 Elements of Digital Citizenship Podcasts

S3 Framework (Safe, Savvy and Social)

Digital Citizenship classifies nine foundational elements in the following three guiding principles: Safe, Savvy and Social (or S3). The tenets of S3 are a way to support, as well as reinforce the framework of the themes of digital citizenship. Each theme/element encompasses three levels of support (Safe, Savvy and Social) which could (or should) be taught as soon as our children can first pick up a device and start to interact with it. The first guiding principal; Safety, focuses on protecting yourself and protecting others and creates the base of digital citizenship. The next is Savvy in which focuses on the concepts around educating yourself and connecting with others. These concepts build upon the concepts of Safety. And finally, the Social guiding principle commits to helping everyone make decisions exemplifying our commitment to respect ourselves and respect others. It is here that we fully realize the possibilities of the online experience.

Safety - Protecting Digital Citizens [being protected from or unlikely to cause danger, risk, or injury to yourself or others]

Technology is changing students coming to our schools which in turn moves education. The tools that have been provided to students and faculty are making tasks such as creating or sharing information (e.g., a document, presentation or video) much more streamlined. While technology affords users new opportunities the issues that occur are often are the lapses of judgment when the tool is not fully understood. These problems happen with moving to new and different tools that knowledge and rules are not fully established by those who use them. This is an attempt to apply older concepts to new ideas, which do not exactly fit. Now is the time to make the shift to how we will be utilizing the technology for the future. It is important that educators now begin making alterations to how technology is viewed and integrated into the classroom. The knowledge we share today will be passed along to the next generation.

Savvy - Creating Educated Digital Citizens [wisdom and practical knowledge; the understanding to make good judgments]

In many school or district mission statements or goals there is often some mention of "being lifelong learners". So what does this mean in the age of digital learning? The growth and changing nature of technology teaches everyone that we can never stop learning. And with the opportunities that these tools provide why would anyone? Technology has opened the possibilities of how to learn new skills and our responsibility to understand and maximize the potential of these tools. There has been an expectation that children innately understand technology since they were born in a time of expanding digital information. Children do show a willingness to use these technologies where adults may approach with more anxiety. It has also been discovered that device and app developers are creating these technologies to make them intuitive, "so that even a child could use." ​

- Educate Yourself/Educate Others

Social - Respecting Yourself as a Digital Citizen [creating cooperative and interdependent relationships and understanding of others]

When discussing the topic of digital citizenship, digital health, digital safety or whatever it may be called in your school or district there are certain universal themes that seem to surface and be at the heart of the issue. Whether it is through topics of cyberbullying, viewing (or posting) inappropriate content, or plagiarism these and other topics of concern that are discussed most among parents and educators. This section geared towards the "social" element defines the general topic of digital citizenship and its main elements of discussion. Humans are social by nature. People choosing to group with others like themselves in cities, states, and countries. As members of a community we tend to connect with those like us. This can be the difficult aspect of trying to interact with others online, everyone is given the opportunity to join this thing called the Internet and even though we try and stay with those most like us, it is almost impossible not to bump into others that want to try and change our minds, our beliefs.

- Respect Yourself/Respect Others

Go to the Progression Chart Page to learn more about implementing these elements into the classroom.

Language Translations of Nine Elements
Portuguese Translation - Nove Elementos da Cidadania Digital

http://www.miudossegurosna.net/nove-elementos-da-cidadania-digital.html (HTML Version)

http://miudossegurosna.net/MSNN-NOVE-ELEMENTOS-DA-CIDADANIA-DIGITAL-v1.pdf (PDF Version)

Italian Translation - I 9 elementi della Cittadinanza Digitale

http://ilgeekinblu.blogspot.it/2014/06/i-9-elementi-della-cittadinanza-digitale.html

# data[].attributes.attachment_link.description
2. Digital Commerce is the electronic buying and selling of goods and focuses on the tools and safeguards in place to assist those buying, selling, banking, or using money in any way in the digital space. Career and technical education use the tools of technology to show students the path for their future.

# data[].attributes.body
Video about a student's story to be used in class for example of effects of cyber bullying.

# data[].attributes.body
Presentation software

# data[].attributes.body
Tool used to communicate with students and parentsz

# data[].attributes.attachment_link.readable_text
Adoption and usage

Simple, effective, and surprisingly easy to use

With nearly 30 million users in 80% of US schools, the numbers speak for themselves: students, families, and educators love using Remind.

# data[].attributes.attachment_link.title
Remind | Communication Platform for Education

# data[].attributes.attachment_link.description
Remind is a communication platform that reaches students and families where they are and supports learning wherever it happens. Millions of educators, students, and parents use Remind to connect with the people and resources that help them teach and learn.

# data[].attributes.body
Google application for creating free websites.

# data[].attributes.attachment_link.description
https://www.youtube.com/watch?v=OsNat-3-D3s Google Sites service to easily create great ...

# data[].attributes.body
Lesson for teaching digital citizenship.

# data[].attributes.attachment_link.description
Common Sense Education provides educators and students with the resources they need to harness the power of technology for learning and life. Find a free K-12 Digital Citizenship curriculum, reviews of popular EdTech apps, and resources for protecting student privacy.

# data[].attributes.body
2018 Washington State Technology Standards

# data[].attributes.body
Free platform for creating easy animated videos.

# data[].attributes.attachment_link.title
Animaker, Make Animated Videos on Cloud for free

# data[].attributes.attachment_link.description
Animaker is an online do-it-yourself (#DIY) animation video maker that brings studio quality presentations within everyone’s reach. Animated Videos, Done Right!

# data[].attributes.body
Website for student practice and mastery.

# data[].attributes.attachment_link.title
Trigonometry : Area Of Circular Sector,Arc of a circle

# data[].attributes.attachment_link.description
Trigonometry : Area Of Circular Sector,Arc Length, Segment of a Circle, Definition and some Examples -iPracticeMath

# data[].attributes.body
This rubric aligns with the Danielson's Framework for Teachers in the way they use technology in the classroom

# data[].attributes.body
Website for graphing, constructing geometric figures, and activities.

# data[].attributes.attachment_link.title
GeoGebra - the world's favorite, free math tools used by over 100 million students and teachers

# data[].attributes.attachment_link.description
Free digital tools for class activities, graphing, geometry, collaborative whiteboard and more

# data[].attributes.body
Website for graphing and activities.

# data[].attributes.attachment_link.description
Desmos offers best-in-class calculators, digital math activities, and curriculum to help every student love math and love learning math.

# data[].attributes.body
Plickers can be used to formatively assess student's knowledge.