# data[].attributes.body
Does It Make a Difference? Evaluating Professional Development

# data[].attributes.attachment_link.readable_text
March 2002 | Volume 59 | Number 6
Redesigning Professional Development Pages 45-51

Thomas R. Guskey

Good evaluations don't have to be complicated. They simply require thoughtful planning, the ability to ask good questions, and a basic understanding of how to find valid answers. What's more, they can provide meaningful information that you can use to make thoughtful, responsible decisions about professional development processes and effects.

What Is Evaluation?

In simplest terms, evaluation is "the systematic investigation of merit or worth"(Joint Committee on Standards for Educational Evaluation, 1994, p. 3). Systematic implies a focused, thoughtful, and intentional process. We conduct evaluations for clear reasons and with explicit intent. Investigation refers to the collection and analysis of pertinent information through appropriate methods and techniques. Merit or worth denotes appraisal and judgment. We use evaluations to determine the value of something-to help answer such questions as, Is this program or activity achieving its intended results? Is it better than what was done in the past? Is it better than another, competing activity? Is it worth the costs?

Some educators understand the importance of evaluation for event-driven professional development activities, such as workshops and seminars, but forget the wide range of less formal, ongoing, job-embedded professional development activities-study groups, action research, collaborative planning, curriculum development, structured observations, peer coaching, mentoring, and so on. But regardless of its form, professional development should be a purposeful endeavor. Through evaluation, you can determine whether these activities are achieving their purposes.

Critical Levels of Professional Development Evaluation

Effective professional development evaluations require the collection and analysis of the five critical levels of information shown in Figure 1 (Guskey, 2000a). With each succeeding level, the process of gathering evaluation information gets a bit more complex. And because each level builds on those that come before, success at one level is usually necessary for success at higher levels.

Figure 1. Five Levels of Professional Development Evaluation

1. Participants' Reactions

Did they like it?

Was their time well spent?

Did the material make sense?

Will it be useful?

Were the chairs comfortable?

2. Participants' Learning

Paper-and-pencil instruments

Simulations

Demonstrations

Participant portfolios

3. Organization Support & Change

District and school records

Minutes from follow-up meetings

Questionnaires

Participant portfolios

To inform future change efforts

Questionnaires

Participant portfolios

Direct observations

Video or audio tapes

5. Student Learning Outcomes

Is student attendance improving?

Are dropouts decreasing?

Student records

School records

Questionnaires

Participant portfolios

Student learning outcomes:

Cognitive (Performance & Achievement)

Affective (Attitudes & Dispositions)

Psychomotor (Skills & Behaviors)

Level 1: Participants' Reactions

The first level of evaluation looks at participants' reactions to the professional development experience. This is the most common form of professional development evaluations, and the easiest type of information to gather and analyze.

At Level 1, you address questions focusing on whether or not participants liked the experience. Did they feel their time was well spent? Did the material make sense to them? Were the activities well planned and meaningful? Was the leader knowledgeable and helpful? Did the participants find the information useful?

Important questions for professional development workshops and seminars also include, Was the coffee hot and ready on time? Was the room at the right temperature? Were the chairs comfortable? To some, questions such as these may seem silly and inconsequential. But experienced professional developers know the importance of attending to these basic human needs.

Information on participants' reactions is generally gathered through questionnaires handed out at the end of a session or activity. These questionnaires typically include a combination of rating-scale items and open-ended response questions that allow participants to make personal comments. Because of the general nature of this information, many organizations use the same questionnaire for all their professional development activities.

Some educators refer to these measures of participants' reactions as "happiness quotients," insisting that they reveal only the entertainment value of an activity, not its quality or worth. But measuring participants' initial satisfaction with the experience can help you improve the design and delivery of programs or activities in valid ways.

Level 2: Participants' Learning

In addition to liking their professional development experience, we also hope that participants learn something from it. Level 2 focuses on measuring the knowledge and skills that participants gained. Depending on the goals of the program or activity, this can involve anything from a pencil-and-paper assessment (Can participants describe the crucial attributes of mastery learning and give examples of how these might be applied in typical classroom situations?) to a simulation or full-scale skill demonstration (Presented with a variety of classroom conflicts, can participants diagnose each situation and then prescribe and carry out a fair and workable solution?). You can also use oral personal reflections or portfolios that participants assemble to document their learning.

Although you can usually gather Level 2 evaluation information at the completion of a professional development activity, it requires more than a standardized form. Measures must show attainment of specific learning goals. This means that indicators of successful learning need to be outlined before activities begin. You can use this information as a basis for improving the content, format, and organization of the program or activities.

Level 3: Organization Support and Change

At Level 3, the focus shifts to the organization. Lack of organization support and change can sabotage any professional development effort, even when all the individual aspects of professional development are done right.

Suppose, for example, that several secondary school educators participate in a professional development program on cooperative learning. They gain a thorough understanding of the theory and develop a variety of classroom activities based on cooperative learning principles. Following their training, they try to implement these activities in schools where students are graded "on the curve"-according to their relative standing among classmates-and great importance is attached to selecting the class valedictorian. Organization policies and practices such as these make learning highly competitive and will thwart the most valiant efforts to have students cooperate and help one another learn (Guskey, 2000b).

The lack of positive results in this case doesn't reflect poor training or inadequate learning, but rather organization policies that undermine implementation efforts. Problems at Level 3 have essentially canceled the gains made at Levels 1 and 2 (Sparks & Hirsh, 1997). That's why professional development evaluations must include information on organization support and change.

At Level 3, you need to focus on questions about the organization characteristics and attributes necessary for success. Did the professional development activities promote changes that were aligned with the mission of the school and district? Were changes at the individual level encouraged and supported at all levels? Were sufficient resources made available, including time for sharing and reflection? Were successes recognized and shared? Issues such as these can play a large part in determining the success of any professional development effort.

Gathering information at Level 3 is generally more complicated than at previous levels. Procedures differ depending on the goals of the program or activity. They may involve analyzing district or school records, examining the minutes from follow-up meetings, administering questionnaires, and interviewing participants and school administrators. You can use this information not only to document and improve organization support but also to inform future change initiatives.

Level 4: Participants' Use of New Knowledge and Skills

At Level 4 we ask, Did the new knowledge and skills that participants learned make a difference in their professional practice? The key to gathering relevant information at this level rests in specifying clear indicators of both the degree and the quality of implementation. Unlike Levels 1 and 2, this information cannot be gathered at the end of a professional development session. Enough time must pass to allow participants to adapt the new ideas and practices to their settings. Because implementation is often a gradual and uneven process, you may also need to measure progress at several time intervals.

You may gather this information through questionnaires or structured interviews with participants and their supervisors, oral or written personal reflections, or examination of participants' journals or portfolios. The most accurate information typically comes from direct observations, either with trained observers or by reviewing video-or audiotapes. These observations, however, should be kept as unobtrusive as possible (for examples, see Hall & Hord, 1987).

You can analyze this information to help restructure future programs and activities to facilitate better and more consistent implementation.

Level 5: Student Learning Outcomes

Level 5 addresses "the bottom line": How did the professional development activity affect students? Did it benefit them in any way? The particular student learning outcomes of interest depend, of course, on the goals of that specific professional development effort.

In addition to the stated goals, the activity may result in important unintended outcomes. For this reason, evaluations should always include multiple measures of student learning (Joyce, 1993). Consider, for example, elementary school educators who participate in study groups dedicated to finding ways to improve the quality of students' writing and devise a series of strategies that they believe will work for their students. In gathering Level 5 information, they find that their students' scores on measures of writing ability over the school year increased significantly compared with those of comparable students whose teachers did not use these strategies.

On further analysis, however, they discover that their students' scores on mathematics achievement declined compared with those of the other students. This unintended outcome apparently occurred because the teachers inadvertently sacrificed instructional time in mathematics to provide more time for writing. Had information at Level 5 been restricted to the single measure of students' writing, this important unintended result might have gone unnoticed.

Measures of student learning typically include cognitive indicators of student performance and achievement, such as portfolio evaluations, grades, and scores from standardized tests. In addition, you may want to measure affective out-comes (attitudes and dispositions) and psychomotor outcomes (skills and behaviors). Examples include students' self-concepts, study habits, school attendance, homework completion rates, and classroom behaviors. You can also consider such schoolwide indicators as enrollment in advanced classes, member-ships in honor societies, participation in school-related activities, disciplinary actions, and retention or drop-out rates. Student and school records provide the majority of such information. You can also include results from questionnaires and structured interviews with students, parents, teachers, and administrators.

Level 5 information about a program's overall impact can guide improvements in all aspects of professional development, including program design, implementation, and follow-up. In some cases, information on student learning outcomes is used to estimate the cost effectiveness of professional development, sometimes referred to as "return on investment" or "ROI evaluation" (Parry, 1996; Todnem & Warner, 1993).

Look for Evidence, Not Proof

Using these five levels of information in professional development evaluations, are you ready to "prove" that professional development programs make a difference? Can you now demonstrate that a particular professional development program, and nothing else, is solely responsible for the school's 10 percent increase in student achievement scores or its 50 percent reduction in discipline referrals?

Of course not. Nearly all professional development takes place in real-world settings. The relationship between professional development and improvements in student learning in these real-world settings is far too complex and includes too many intervening variables to permit simple causal inferences (Guskey, 1997; Guskey & Sparks, 1996). What's more, most schools are engaged in systemic reform initiatives that involve the simultaneous implementation of multiple innovations (Fullan, 1992). Isolating the effects of a single program or activity under such conditions is usually impossible.

But in the absence of proof, you can collect good evidence about whether a professional development program has contributed to specific gains in student learning. Superintendents, board members, and parents rarely ask, "Can you prove it?" Instead, they ask for evidence. Above all, be sure to gather evidence on measures that are meaningful to stakeholders in the evaluation process.

Consider, for example, the use of anecdotes and testimonials. From a methodological perspective, they are a poor source of data. They are typically highly subjective, and they may be inconsistent and unreliable. Nevertheless, as any trial attorney will tell you, they offer the kind of personalized evidence that most people believe, and they should not be ignored as a source of information. Of course, anecdotes and testimonials should never form the basis of an entire evaluation. Setting up meaningful comparison groups and using appropriate pre- and post-measures provide valuable information. Time-series designs that include multiple measures collected before and after implementation are another useful alternative.

Keep in mind, too, that good evidence isn't hard to come by if you know what you're looking for before you begin. Many educators find evaluation at Levels 4 and 5 difficult, expensive, and time-consuming because they are coming in after the fact to search for results (Gordon, 1991). If you don't know where you are going, it's very difficult to tell whether you've arrived. But if you clarify your goals up front, most evaluation issues fall into place.

Working Backward Through the Five Levels

Three important implications stem from this model for evaluating professional development. First, each of these five levels is important. The information gathered at each level provides vital data for improving the quality of professional development programs.

Second, tracking effectiveness at one level tells you nothing about the impact at the next. Although success at an early level may be necessary for positive results at the next higher one, it's clearly not sufficient. Breakdowns can occur at any point along the way. It's important to be aware of the difficulties involved in moving from professional development experiences (Level 1) to improvements in student learning (Level 5) and to plan for the time and effort required to build this connection.

The third implication, and perhaps the most important, is this: In planning professional development to improve student learning, the order of these levels must be reversed. You must plan "backward" (Guskey, 2001), starting where you want to end and then working back.

In backward planning, you first consider the student learning outcomes that you want to achieve (Level 5). For example, do you want to improve students' reading comprehension, enhance their skills in problem solving, develop their sense of confidence in learning situations, or improve their collaboration with classmates? Critical analyses of relevant data from assessments of student learning, examples of student work, and school records are especially useful in identifying these student learning goals.

Then you determine, on the basis of pertinent research evidence, what instructional practices and policies will most effectively and efficiently produce those outcomes (Level 4). You need to ask, What evidence verifies that these particular practices and policies will lead to the desired results? How good or reliable is that evidence? Was it gathered in a context similar to ours? Watch out for popular innovations that are more opinion-based than research-based, promoted by people more concerned with "what sells" than with "what works." You need to be cautious before jumping on any education bandwagon, always making sure that trustworthy evidence validates whatever approach you choose.

Next, consider what aspects of organization support need to be in place for those practices and policies to be implemented (Level 3). Sometimes, as I mentioned earlier, aspects of the organization actually pose barriers to implementation. "No tolerance" policies regarding student discipline and grading, for example, may limit teachers' options in dealing with students' behavioral or learning problems. A big part of planning involves ensuring that organization elements are in place to support the desired practices and policies.

Then, decide what knowledge and skills the participating professionals must have to implement the prescribed practices and policies (Level 2). What must they know and be able to do to successfully adapt the innovation to their specific situation and bring about the sought-after change?

Finally, consider what set of experiences will enable participants to acquire the needed knowledge and skills (Level 1). Workshops and seminars, especially when paired with collaborative planning and structured opportunities for practice with feedback, action research projects, organized study groups, and a wide range of other activities can all be effective, depending on the specified purpose of the professional development.

This backward planning process is so important because the decisions made at each level profoundly affect those at the next. For example, the particular student learning outcomes you want to achieve influence the kinds of practices and policies you implement. Likewise, the practices and policies you want to implement influence the kinds of organization support or change required, and so on.

The context-specific nature of this work complicates matters further. Even if we agree on the student learning outcomes that we want to achieve, what works best in one context with a particular community of educators and a particular group of students might not work as well in another context with different educators and different students. This is what makes developing examples of truly universal "best practices" in professional development so difficult. What works always depends on where, when, and with whom.

Unfortunately, professional developers can fall into the same trap in planning that teachers sometimes do-making plans in terms of what they are going to do, instead of what they want their students to know and be able to do. Professional developers often plan in terms of what they will do (workshops, seminars, institutes) or how they will do it (study groups, action research, peer coaching). This diminishes the effectiveness of their efforts and makes evaluation much more difficult.

Instead, begin planning professional development with what you want to achieve in terms of learning and learners and then work backward from there. Planning will be much more efficient and the results will be much easier to evaluate.

Making Evaluation Central

A lot of good things are done in the name of professional development. But so are a lot of rotten things. What educators haven't done is provide evidence to document the difference between the two.

Evaluation provides the key to making that distinction. By including systematic information gathering and analysis as a central component of all professional development activities, we can enhance the success of professional development efforts everywhere.

References

Fullan, M. G. (1992). Visions that blind. Educational Leadership, 49(5) 19-20.

Gordon, J. (1991, August). Measuring the "goodness" of training. Training, 19-25.

Guskey, T. R. (1997). Research needs to link professional development and student learning. Journal of Staff Development, 18(2), 36-40.

Guskey, T. R. (2000a). Evaluating professional development. Thousand Oaks, CA: Corwin.

Guskey, T. R. (2000b). Grading policies that work against standards and how to fix them. NASSP Bulletin, 84(620), 20-29.

Guskey, T. R. (2001). The backward approach. Journal of Staff Development, 22(3), 60.

Guskey, T. R., & Sparks, D. (1996). Exploring the relationship between staff development and improvements in student learning. Journal of Staff Development, 17(4), 34-38.

Hall, G. E., & Hord, S. M. (1987). Change in schools: Facilitating the process. Albany, NY: SUNY Press.

Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards(2nd ed.). Thousand Oaks, CA: Sage.

Joyce, B. (1993). The link is there, but where do we go from here? Journal of Staff Development, 14(3), 10-12.

Parry, S. B. (1996). Measuring training's ROI. Training & Development, 50(5), 72-75.

Sparks, D. (1996, February). Viewing reform from a systems perspective. The Developer, 2, 6.

Sparks, D., & Hirsh, S. (1997). A new vision for staff development. Alexandria, VA: ASCD.

Todnem, G., & Warner, M. P. (1993). Using ROI to assess staff development efforts. Journal of Staff Development, 14(3), 32-34.

Thomas R. Guskey ( Guskey@uky.edu) is a professor of education policy studies and evaluation, College of Education, University of Kentucky, Taylor Education Building, Lexington, KY 40506, and author of Evaluating Professional Development (Corwin Press, 2000).

# data[].attributes.attachment_link.title
Does It Make a Difference? Evaluating Professional Development

# data[].attributes.attachment_link.description
March 2002 | Volume 59 | Number 6 Redesigning Professional Development Pages 45-51 Thomas R. Guskey Good evaluations don't have to be complicated. They simply require thoughtful planning, the ability to ask good questions, and a basic understanding of how to find valid answers.

# data[].attributes.body
Webinar: Assessing the Impact of Professional Development

# data[].attributes.attachment_link.readable_text
Facilitated by Joellen Killion

Watch this webinar.

Get complete webinar archive access. Join us today!

Learning Forward Senior Advisor Joellen Killion facilitates a discussion designed to help school and district leaders plan and conduct an evaluation of professional learning. Develop strategies to assess a professional development program for its degree of evaluability, and to construct an evaluation framework - including a program's theory of change, logic model, evaluation questions, data sources, data collection methods, data analysis plan, timelines, and persons responsible -- that outlines a detailed plan for an evaluation.

As you watch this webinar, consider the following questions:

What is the purpose of assessing the impact of professional learning?

How do you construct a framework that outlines a detailed plan for evaluation?

How do you tie your professional learning objectives to measurable short-, medium- and long-term results for educators and students?

Joellen Killion is Senior Advisor for Learning Forward. As Deputy
Executive Director of Learning Forward, she led a number of
initiatives related to examining the link between professional
development and student learning, and led the revision process for
the new Standards for Professional Learning. She has extensive
experience in professional development planning, design,
implementation, and evaluation both at the school and system level.
Her books include Assessing Impact: Evaluating Staff Development; Taking the Lead: New Roles for Teacher and School-based Coaches (with Cindy Harrison); The Learning Educator: A New Era in Professional Learning (with Stephanie Hirsh); and Becoming a Learning School (with Patricia Roy).

# data[].attributes.attachment_link.description
Facilitated by Joellen Killion Watch this webinar. Get complete webinar archive access. Join us today! Learning Forward Senior Advisor Joellen Killion facilitates a discussion designed to help school and district leaders plan and conduct an evaluation of professional learning.

# data[].attributes.body
Video: Pineapple Charts

# data[].attributes.body
Building Teacher Collaboration School-wide

# data[].attributes.attachment_link.provider_name
AMLE - Association for Middle Level Education

# data[].attributes.attachment_link.readable_text
Teacher collaboration has been a common element of middle grades initiatives for years, typically one or more of these three organizational models: common planning time, professional learning communities, and critical friends groups.

Each model is distinct, yet they share common features. They 1) advance teacher learning, 2) address context-specific issues, 3) foster collegiality, 4) reduce teacher isolation, and 5) lead teachers to greater insights about teaching and learning. The overarching, and arguably the most important common element, is the goal of improved student learning.

What makes each organization model unique? First, the teachers are organized differently in each model: interdisciplinary teams, disciplinary teams, or self-selected teams. Second, the starting point for teachers' collaboration differs. In common planning time, teacher teams begin with an analysis of the holistic needs of students; in professional learning communities, the teacher teams begin with the analysis of students' academic progress; and in the critical friends groups, the teacher teams begin with an identified need or interest for improved practice. The recognized features of each organization model are summarized in Table 1.

Table 1. Organizatonal Models That Promote Teacher Collaboration

Common Planning Time Professional Learning Communities Critical Friends Groups

Interdisciplinary teams- teachers share same students

Coordinate team policies and procedures

Discuss students

Meet with parents

Plan team activities, thematic or cross-curricular units

Examine student work

Participate in professional development

Disciplinary teams

Ongoing process of collective inquiry and action research

Collective analysis of student assessment data in relation to specific learning targets

Use of data to inform and assess effectiveness of instruction

Group gathers voluntarily to improve practice through collaborative learning

Uses coaches and specific protocols used to guide sessions

Identify school-specific student learning goals, reflect on practices for achieving the goals, collaboratively examine student work

Strategies for Building Teacher Collaboration

Organizational models facilitate, but do not guarantee collaboration. How teachers engage in a model can make a difference. Teachers' personal stance about whether they "have to" or "want to" participate in an organizational model is critical to successful collaboration. Equally important is understanding how to engage effectively in collaborative work with colleagues.

As with other skills, we gain a greater capacity for collaboration with the opportunity to practice. To initiate or revitalize teacher collaboration in your school, try these five strategies.

1. Create a truly shared vision and goals. The level of ownership they feel in the process influences how much teachers actually invest in collaborative work. A shared vision and goals can lead to that sense of ownership. For example, identify your team's shared vision of caring for students and student learning, set goals related to that vision, discuss how the team's work can help attain those goals, and check in often to assess progress. The strong connection between the work and the vision of the team can help individuals see purpose and assume ownership in the process.

2. Develop a sense of community. At its core, collaboration is relational. Getting to know your colleagues, understanding their passions, and taking the time to connect on a personal level can help members gain mutual respect and look past perceived eccentricities in others. Establishing shared values and commitments can unify the group and provide purpose for their collective work. Like all relationships, a collaborative community develops over time and requires work to maintain.

Trust influences the effectiveness of collaborative work. Respecting group commitments such as being fully present at meetings and seeing the best in others helps establish trust and build a cohesive community. Other ways to develop community include establishing traditions, celebrating accomplishments, and recognizing individual contributions.

3. Identify group norms. Let's face it: collaboration can be uncomfortable or stressful at times. When we are transparent about our work and our beliefs, our colleagues can see our limitations as well as our strengths, placing us in a position of vulnerability. Sharing with and trusting colleagues requires courage and humility. A climate of trust can help establish the safe environment that's necessary for open communication.

Identifying and establishing group norms also can help develop that safe environment. Norms might include defining roles and responsibilities, using protocols for interpersonal communication, and outlining parameters for time management.

Taking the time to get to know the learning styles, needs, interests, fears, and hopes of each team member helps shape the norms for how the group engages in the shared work.

4. Use discussion and dialogue. Whether they are integrating curriculum, analyzing data, or studying a new practice, teams should understand the roles of, and differences between, dialogue and discussion. They are equally important to the group process.

Discussion moves the conversation forward. In discussion, individuals state their opinions for the purpose of building consensus or making decisions.

The goal of dialogue is to share and broaden knowledge. Dialogue invites multiple perspectives, values the exploration of biases and assumptions, questions the status quo, and entertains new ways of knowing and being. Dialogue requires active listening, willingness to state beliefs, the ability to bear the tension of ambiguity, and belief in the transformative potential in the process.

5. Work through conflict. Dialogue can cultivate deep professional learning as individuals and teams explore new ideas for practice. However, dialogue may also lead to conflict. It can be helpful for your team to develop a conflict management plan and to monitor conflict as it arises.

Teams can help manage conflict by providing time, space, grace, and support for individuals as they work through their emotions. Individuals also should monitor their own emotions and practice self-care.

Using professional judgment, your team can determine when to explore the roots of conflict and when to provide space for reflection and cooling down. While sometimes uncomfortable, conflict often provides growth opportunities.

Teacher Collaboration

Strong collaboration and collaborative cultures develop over time and require commitment to the process. While the benefits are clear, genuine collaboration is complex.

Patience in the moment and anticipation for the outcome can lead to deep teacher learning that translates into tangible student achievement.

What will it take to maximize organizational models for productive teacher collaboration in your school? School leaders-principals and teachers-need to work together and commit to a collaborative culture. They need to ensure dedicated time for the organizational model within the school day.

Common planning time, professional learning communities, and critical friends groups each require regular, dedicated time for teachers to collaborate. With time, teachers can develop authentic collaborative communities in which they address common issues, shared goals or school-wide initiatives; engage in mutually beneficial endeavors using communal resources; and advance their skills, knowledge, and dispositions related to student learning.

Micki M. Caskey is the associate dean for academic affairs in the Graduate School of Education at Portland State University. She is past chair of the Association of Middle Level Education's Research Advisory Committee and past editor of Research in Middle Level Education Online. caskeym@pdx.edu

Jan Carpenter is chair of the Education Department in the School of Graduate Studies at Marylhurst University. She is also chair of the Association of Middle Level Education's Professional Preparation Advisory Committee and past chair of the Oregon Middle Level Consortium. carpenter@marylhurst.edu

Published in AMLE Magazine, October 2014.

# data[].attributes.attachment_link.description
Teacher collaboration has been a common element of middle grades initiatives for years, typically one or more of these three organizational models: common planning time, professional learning communities, and critical friends groups. Each model is distinct, yet they share common features.

# data[].attributes.body
Evaluating Professional Development

# data[].attributes.attachment_link.readable_text
Joellen Killion from the National Staff Development Council outlines an eight-step process for measuring the impact of professional development.

For many years, the only evaluation of professional development was the traditional end-of-program "smiley" sheet on which participants reported their degree of satisfaction with the program, presenter, and facilities. Policy and decision makers who wanted to know if professional development produced any results had few options.

That changed in the mid-1990s, when sweeping changes in federal policy required those spending federal funds to evaluate the effectiveness of professional development. Many professional development specialists recognized the weakness of their evaluation attempts, and some argued that it was impossible to link professional development with student achievement because of the large number of intervening variables. Others wondered if the evaluation field provided the necessary tools and processes to measure impact of professional development.

In 1999, with a grant from the Edna McConnell Clark Foundation, we at the National Staff Development Council (NSDC) launched a 2-year initiative to find ways to measure the impact of professional learning on teacher behavior and student learning. With a team of experts in evaluation and professional development, we discovered that the major problem with evaluating professional development lay not in evaluation but in the design of professional development. Educators wrongly believed that one-shot professional development sessions would transform not only teacher classroom behavior but also student learning. Confronting this fallacy presented a new challenge for professional development leaders and providers: If one-shot sessions do not work, what does it take to change teacher classroom behavior and student learning?

The answer was that it almost always takes more than just a single session. Ongoing sessions of learning, collaboration, and application, accompanied by school- and classroom-based support, over an ample time period are necessary to incorporate new behaviors fully into a teacher's repertoire. If the design of professional development is sufficiently strong and long enough to promote deep changes, it will be possible to measure the impact of professional development on student learning.

Using a theory of change evaluation model and building on logic models that define the transformation process, we developed an eight-step evaluation process that encourages evaluators to build pathways with evidence to measure the impact of professional development on teacher classroom behavior and student learning.
An Eight-Step Process for Measuring Impact
1. Assess evaluability. Evaluators examine the design of the professional development program to determine its likelihood of producing the intended results; scrutinize the program's goals, objectives, standards of success, indicators of success, theory of change, and logic model; and ask about the program's clarity, feasibility, strength, and worth. If, after that analysis, the program is deemed evaluable, the evaluator moves ahead to Step 2. If the program is deemed not evaluable, the evaluator encourages the program's designer(s) to revise the program.

2. Formulate evaluation questions. Evaluators design the formative and summative questions, which focus on the initial and intermediate outcomes and the program's goals and objectives. By asking questions about results (e.g., did teachers use the strategies? did student work demonstrate evidence of teachers' application of the strategies?) rather than about services, evaluators can measure impact rather than program delivery.

3. Construct the evaluation framework. Evaluators determine what evidence to collect, from whom or what sources to collect the evidence, how to collect the evidence, and how to analyze the evidence.

4. Collect data. Evaluators use the data collection methods determined in Step 3 to collect evidence to answer the evaluation questions.

5. Organize and analyze data. Evaluators organize and analyze collected data and display analyzed data in multiple formats to use in Step 6.

6. Interpret data. Working together, stakeholders and evaluators interpret the data to make sense of it, draw conclusions, assign meaning, and formulate recommendations. Including stakeholders in this process is essential because their participation expands and enhances the meaning of the data.

7. Report findings.Evaluators report findings and make recommendations in formats sensitive to the needs of the multiple audiences. Rather than a single technical report, evaluators prepare multiple reports of varied lengths and in varied levels of sophistication and formats.

8. Evaluate the evaluation. The evaluator analyzes his or her own evaluation methodology, processes, resources, skills, and so forth. As a reflective practitioner, the evaluator looks back at the work done and identifies its strengths and areas for continued refinement and growth.

In addition to using this eight-step process, it is essential that evaluators believe that the professional development program has the potential to produce the intended results. Lack of belief in professional development's potential-not evaluation-has been the greatest challenge in evaluating professional development.

A theory of change identifies the sequence of actions a program intends to take to accomplish its goals and the assumptions upon which those actions rest.
A logic model is a tool that turns the actions into results, so that evaluators and program managers can monitor progress and results.
Formative evaluations are conducted during program implementation in order to provide information that will strengthen or improve the program being studied.
Summative evaluations are conducted either during or at the end of a program's implementation. They determine whether a program's intended outcomes have been achieved.

Joellen Killion
Director of Special Projects
National Staff Development Council
10931 W. 71st Place
Arvada, CO 80004-1337
Tel: 303-432-0958
Fax: 303-432-0959
Email: joellen.killion@nsdc.org

# data[].attributes.attachment_link.keywords[]
hfrp harvard family research project evaluation exchange family involvement school parent children

# data[].attributes.attachment_link.keywords[]
early childhood education strategic partnership out-of-school complementary learning

# data[].attributes.attachment_link.title
Evaluating the Impact of Professional Development in Eight Steps

# data[].attributes.attachment_link.description
Joellen Killion from the National Staff Development Council outlines an eight-step process for measuring the impact of professional development. For many years, the only evaluation of professional development was the traditional end-of-program "smiley" sheet on which participants reported their degree of satisfaction with the program, presenter, and facilities.

# data[].attributes.body
Podcast: PD That Works

# data[].attributes.attachment_link.title
PD That Works: Guskey's 5 Levels of PD Evaluation from Level Up Learning

# data[].attributes.attachment_link.description
Listen to episodes of Level Up Learning on podbay.fm.

# data[].attributes.body
Pineapple Chart

# data[].attributes.attachment_link.readable_text
This post contains Amazon Affiliate links. If you click these and make a purchase from Amazon, I will receive a small commission at no extra cost to you. Thanks for your support.

Professional Development: The phrase has a way of striking dread into the hearts of teachers. But not because teachers don't WANT to improve. Not because we believe we're done growing and learning. Far from it.

In fact, so many teachers bristle at the thought of PD because most of the time, it's executed so poorly. Although the typical one-size-fits-all format-where every teacher in the building is herded together to listen to an expert speak-has been widely denounced, it still persists as the default model. Sometimes, if the speaker happens to be engaging, some teachers will walk away with a small tidbit they can apply to their own work. For most teachers, though, it feels like a waste of precious time, time that could be spent developing skills that would make a real difference in their specific practice.

But things are starting to change. More and more schools are experimenting with personalized approaches to professional development; this guide from EdSurge takes a comprehensive look at all the different shapes this can take. And as new unconferences pop up every week all over the world, teachers are seeing just how easy it can be to learn from each other, without the need for any kind of outside expert.

And now, taking PD to an even simpler, more local, more affordable level, we have the Pineapple Chart.

What is a Pineapple Chart?

A Pineapple Chart is a system that allows teachers to invite one another into their classrooms for informal observation. The chart is set up in some location where teachers go on a daily basis: the teacher's lounge, the copy room, or wherever teacher mailboxes live in your school. On the chart, teachers "advertise" the interesting things they are doing in their classrooms, activities they think others might want to observe. The activities could be as complex as a science lab, a history simulation, or a Skype session with a school in another country. Or they could be as simple as a read-aloud or a lesson on badminton.

The chart represents one week of school. Along the top, five columns are labeled Monday through Friday. Along the side, rows assigned to various chunks of each school day. In a middle or high school, these would be class periods. In an elementary school, the rows could be divided by hours or half-hours.

When a teacher sees something on the chart she (or he) is interested in, she goes to that classroom at the designated time, sits down in an out-of-the-way spot, and watches. That's it. No note-taking is required, no post-observation conference, no write-up. Just a visit. She can even grade papers or catch up on email if she wants, paying closer attention when the moment calls for it, but getting work done in the meantime. She can stay for five minutes or a whole class period. The key word here is informal, and it's the best way for teachers to learn lots of skills and techniques just when they need them.

The reasons the visiting teacher might come in are endless: It could be that she wants to start using Google Cardboard, but would like to see another teacher do it first. Maybe she's struggling with classroom management and knows this teacher runs a tight ship. Or maybe she's there because she always wanted to learn more about World War I. Regardless of the initial reason, what she ends up learning is likely to be far more than she expected. During that Google Cardboard visit, she'll also notice a cool bulletin board in that teacher's classroom and decide to try something similar in her room. While watching the lesson on World War I, she'll pick up a questioning technique she ends up using in her own teaching. Every teacher in your building has their own unique blend of skills and talents, but the only way you'll ever learn about them is to see your colleagues in action.

The Origins of the Pineapple Chart

The pineapple has long been a symbol of hospitality; it's used in the chart in the spirit of welcoming one another into our school "homes."

But that's just the beginning. Here's how the Pineapple Chart came to be: In early 2015, Mark Barnes and I were compiling innovative teaching "hacks" for our book, Hacking Education: 10 Quick Fixes for Every School. I remembered my sister, who teaches science in Massachusetts, telling me about her attempts to start a "Pineapple Welcome" initiative at her school, where teachers would hang a picture of a pineapple outside their classroom doors, indicating that any other teacher who wanted to visit was welcome to stop by.

When I asked her how the whole pineapple thing went, she said, "Eh. Some people came in, but usually I would hang the pineapple and nothing would happen. I stopped doing it after a while."

This bothered me, because some of the most valuable skills I have picked up as a teacher came from informally observing my colleagues in their classrooms. I feel strongly that some of the best professional development available to teachers lives right inside the walls of our schools, and if we could just watch each other teach more, we would all grow exponentially.

Just hanging the pineapple on the door, I realized, had two shortcomings. One, it was only visible to teachers who happened to pass by that door on that day. And two, it didn't offer enough information: Just seeing a pineapple on the door tells the visitor nothing about what's going on inside. If a teacher is going to spend time visiting another teacher's classroom, he wants to know that the thing he's coming to observe is something he's actually interested in. A chart posted in a central location, with a bit of information about the day's activities, solves both problems.

We asked my sister to pilot this centralized system at her school, and the first Pineapple Chart was born. Since the publication of Hacking Education, the Pineapple Chart system has taken off in schools all across the world, and the results have been fantastic.

Two Schools Where Pineapple Charts Are Making an Impact

Gator Run Elementary, Weston, Florida

"The Pineapple Chart has been a game changer for the staff at Gator Run," says Intern Principal Cyndi Felton, who initiated Gator Run's first Pineapple Chart in January of 2016. "We have some of the very best PD available under our own roof. Now teachers have the opportunity to get into each other's classrooms and learn from one another."

Since the chart has been going, Felton has seen teachers trying new things more quickly. "Last year we noticed tools and strategies spreading throughout the building after Pineapple Chart visits. Kahoot, Nearpod, Plickers, in-class flipped lessons, SeeSaw, and performance scales in addition to unique procedures, classroom management and teaching strategies."

Maple Avenue Middle School, Littlestown, Pennsylvania

In January of 2016, when Assistant Principal Cortney Golden first launched the chart in her school, she worked hard to keep it full. "I was very aware that if we lost momentum, and the chart was blank, it would die. I sent staff-wide emails, group emails, personal emails, face to face requests, and even had my secretary appealing to teachers to sign up. Getting teachers to sign up was my first priority....then, once that became more automatic, we began to 'push' visiting."

Since then, informal observation and an open-door policy have become part of the school's culture. So much so that the word "pineappling" has become a verb:

"Collectively, we embrace pineappling now more than ever," says Golden. "As teachers desire to grow their craft, to foster 21st century skills, and to help students discover and pursue their passions, it becomes clear to all that teachers need each other. Pineappling gives teachers the permission they feel they need to collaborate, observe, ask, and reflect. It is a 'safe' and powerful professional development opportunity. Pineappling is free, fits within the school day, and has proven to be the BEST professional growth experience."

Tips for Making Pineapple Charts Work

Find creative ways to make the time.
The simplest way to squeeze these visits in is for teachers to do them during their planning time, or part of it. Remember, we're not talking about a whole class period necessarily; a 15-minute visit may be all that's needed to pick up a new strategy. In cases where teachers want to see something that doesn't coincide with planning time, the staff will need to be more creative:

A floating sub can be arranged for a day (or a few days) and teachers can "sign up for the sub" in order to visit another room.

Administrators (or other certified personnel) can step in to cover a class. This solution does double-duty, as it gives the principal some time to interact more closely with small groups of students. Instead of covering the teacher's regular material, the administrator could use this time for having discussions about school issues or just getting to know the students.

Teachers can team up to cover each other's classes. Imagine a scenario where teacher A agrees to cover teacher B's class while teacher B observes in teacher C's room. Then everyone switches roles two more times. A motivated school might even experiment with a second chart (beside the Pineapple Chart) where teachers post a need for coverage and other teachers sign up to cover for them.

Keep it voluntary.
Administrators, be warned! The quickest way to suck the life out of this initiative is to make it mandatory for all teachers. By keeping visits informal and strictly voluntary (for the host teacher AND the visitor), you'll ensure that the Pineapple Chart is always seen as a positive thing, a learning opportunity initiated by the teacher.

Skip the documentation.
An equally fast way to drain people's enthusiasm for visits is to force them to do some kind of write-up afterward. Please don't turn the chart into another reason to require teachers to document. If they know paperwork is expected of them, they will be far less likely to visit.

Start with the bold and the brave...
When you first launch the Chart, many teachers will be shy about posting lessons. Get things going by having your most extroverted teachers sign up first. If you get a group of 5-10 teachers willing to put themselves out there, this will eventually help others step forward as well.

...but set the bar low.
Make sure teachers know they don't have to be doing something 'impressive' in order to post a lesson to the chart. "The hardest part is getting teachers to understand it doesn't have to be an extravagant lesson to sign up," says Felton. "Their everyday strategies and standard procedures can be incredibly helpful to someone who has never seen or thought of it. You really never know what you might learn whether it's a 5- or a 25-minute visit."

Sell it.
Especially when you first get started, encourage those who do the observing to share the good things they saw by talking about it in faculty meetings, sharing a compliment on a teacher "brag board" or posting about it online. At Maple Avenue, Golden encouraged teachers to tweet about their experiences using the hashtag#MAMSLearning. "This helped keep the excitement high, and the pressure low," she says. "Although I still have to harass teachers every once and again (I call it "gently wooing") to showcase the teaching and learning in their classroom, it has become very much part of our culture."

Share Your Story

If your school has already started using a Pineapple Chart, tell us about it in the comments. I would love to hear about tweaks you've made to the system and how the system is impacting teacher relationships and student growth.

Learn More

The Pineapple Chart is just one of the 10 hacks Mark Barnes and I share in our 2015 book, Hacking Education: 10 Quick Fixes for Every School. Get your copy today and start hacking tomorrow! ♦

Learn something new every week.
Join my mailing list and get weekly tips, tools, and inspiration all geared toward making your teaching more effective and fun. You'll also get access to my members-only library of free resources, including my e-booklet, 20 Ways to Cut Your Grading Time in Half, which has helped thousands of teachers spend less time grading!

# data[].attributes.attachment_link.title
How Pineapple Charts Revolutionize Professional Development

# data[].attributes.attachment_link.description
As one-size-fits-all PD fades into obscurity, more personalized professional development models have emerged. The Pineapple Chart may be the greatest of them all.

# data[].attributes.headline
Summary of Content Standard 3 and Process Standard 8

# data[].attributes.subject
Summary of Content Standard 3 and Process Standard 8

# data[].attributes.body
Video: Teacher Labs: Making Professional Development Collaborative

# data[].attributes.attachment_link.title
Teacher Labs: Making Professional Development Collaborative