# data[].attributes.body
If you want to impact student learning or educator effectiveness, you need to have a plan for evaluating Professional learning.

# data[].attributes.body
Step 1 - new take that was unfamiliar

"If the design pf PD is sufficiently strong and long enough to promote deep changes, it will be possible to measure the impact of PD on student learning." - requires ongoing learning sessions, collaboration, application, and school- and classroom support.

8 steps are comprehensive. Nice to have a systematic process.

# data[].attributes.body
Providing nano learning chunks (maybe ~30 minutes) that reoccurs consistent for a set time period (such as once every two weeks).

# data[].attributes.body
Pre and Post assessments of the districts; pre and post assessment of the coaches. What non academic measures can be assessed (Graduation rate, disproportionality) etc. Leverage pre-assessent questions- immediate workshop registration- then a month later, ask for stories, impact, reflection on their hopes for the session.

# data[].attributes.body
Want to be Going beyond the feedback.

Not looking at beneficiaries.

Going well in MTSS conversations but not necessarily other initiatives

Sometimes districts just want the quick training and nothing further

Maybe when taking that request perhaps asking about district expectations

When reporting results and findings of data tailor to audience.

Now what? Being more intentional with design to build that relationship with school.

# data[].attributes.body
2) We have to be intentional about sharing our professional learning journeys with parents and community so we are modeling life-long learning and professional growth and so the parents/community are more understanding when schedule changes are needed to accommodate professional learning for staff.

# data[].attributes.body
Evaluations aren't fully measurable on single-session PDs.

# data[].attributes.body
1) It's important to be intentional about sharing the impact of professional learning and the stories associated with it with the School Board. This helps to create support for professional learning.

# data[].attributes.body
Does using modules in online CPE courses allow for more repetition in learning that makes it not just a single-session PD?

# data[].attributes.body
Paragraph 4: "Ongoing sessions of learning, collaboration, and application, accompanied by school- and classroom-based support, over an ample time period are necessary to incorporate new behaviors fully into a teacher's repertoire."

# data[].attributes.body
Paragraph 3: "Educators wrongly believed that one-shot professional development sessions would transform not only teacher classroom behavior but also student learning."

# data[].attributes.body
Reflect on what data is important to collect (Number of participants? Number of districts represented? District/school needs based on goals?)

# data[].attributes.body
Q1:

TaC has a service log and after each touchpoint the consultant logs time and effort.

CRM - Sales Force (capture data in many different ways, expensive, but want to bring it to the IU.

STaR - to track time TaC

# data[].attributes.body
Considering the investment in PD (many staff members X many sessions X much time = well…$$$)

so, is it irresponsible to carry this out without measurement?

And if student growth is the real purpose, how do we do research design to structure this?

Almost need a program evaluator to help with this

Some IUs have a program eval team, but this is usually focused on larger grant projects (more external)

Metrics we want to look at:

Setting learning goals + how do we measure something like that?

Sometimes we set up artifacts that will demonstrate learning goals

# data[].attributes.body
Debrief with participants about what worked and what didn't, especially after big events. Struggle to keep up with it for the one day workshops.

# data[].attributes.body
Building relationships that people feel comfortable sharing constructive criticisms

# data[].attributes.body
Data coordinator pulls information and reports to executive director.

# data[].attributes.body
Looking at surveys, especially questions involving constructive criticism, to see where there is room for improvement

# data[].attributes.body
What is worth implementing?

How will teachers use PD content in their classrooms?

What are next steps/next requests?

# data[].attributes.body
PD Reflection form for staff developers to complete (What went well? What would you change?)

# data[].attributes.body
Pre-survey and post-survey

# data[].attributes.body
How do we leverage both storytelling AND numbers? There are many different ways that revenue can be used as a metric, but the make up of each IU is different and our needs and goals are as unique as the people we serve.

# data[].attributes.body
Number of attendees

# data[].attributes.body
Enrollment, Feedback from participants, Informal interviews

# data[].attributes.body
Returning participants and people attending based on referrals

# data[].attributes.body
Metrics: quantity (number of trainings), quality of trainings, student performance and improvements

# data[].attributes.body
Focus groups for reflection, Act 48 evaluation surveys, qualitative comments, walkthrough observations for those who supervise staff, reflective conversations

# data[].attributes.body
Teacher's usage and application of the skills gained in workshops

# data[].attributes.body
Sending follow up surveys a year or so later

# data[].attributes.body
Track data by: topic, school, tac vs curriculum, atsi, external staff vs. Internal staff, system of support, statement of work on training input form

Evaluation after PL session completed by participants

# data[].attributes.body
Surveys associated for accountability- challenge with honesty, length of surveys,

How do we help people to come back with stories? What would it look like to come back 6 months later to share a story of impact/success.

# data[].attributes.body
Track amount of people attending sessions.

# data[].attributes.body
enrollment, cancellations, surveys, evaluations, interviews from participants, feedback surveys aligned to performance goals

# data[].attributes.body
Our PD sessions may be on an ongoing assignment basis or ad hoc.

easier to gather data when PD is completed on an ongoing basis.

^Allows for a benchmark and change monitoring

In either case, above, there’s still a challenge of attribution—how do we know our PD was responsible for an outcome (as opposed to something a teacher did or something else entirely)

—-

Also do tons of PD for own staff, but do little data collection in terms of quality monitoring

Surveys used in the past felt somewhat tokenized—doing them just to do it

Can we do multiple regression analysis on our PD delivery?

# data[].attributes.body
PDE's survey

# data[].attributes.body
combination of Likert scale questions and narrative responses following pd sessions

# data[].attributes.body
Informal collection during pre- and post-observations, then discuss in meetings.

# data[].attributes.body
We use a professional learning survey following every session delivered. All the information is collected in a Google sheet for the individual presenter to peruse and can be analyzed collectively as well.

# data[].attributes.body
Evaluations after sessions

# data[].attributes.body
Custom developed tracker for the more business ended data.

# data[].attributes.body
Custom survey that mirrors PDE's survey that includes a few more open ended questions at the end of it.

# data[].attributes.attachment_link.provider_name
Learning Forward - The Professional Learning Association

# data[].attributes.attachment_link.readable_text
To make informed decisions about professional learning, educators use data gathered from formative and summative evaluations. In an era of rapid-cycle change as is common with continuous improvement within communities of practice, school improvement, or cycles of inquiry, evaluation becomes a core part of the process. Assessing the impact of well-designed professional learning as well as the enactment of the learning experiences is crucial. Too often the mere act of checking off completed actions is considered a substitution for or approximation of the intended results. In other words, the plan for professional learning or its implementation is not the same as its effects.
Effectiveness¹ and effectiveness²
The evaluation of professional learning measures the effectiveness or quality of the learning experience (effectiveness ¹) and impact or results of the learning program (effectiveness ²). If for example, district leaders want to evaluate a district's coaching program, the evaluation may focus on the implementation of coaching, how coaches spend their time, how much coaching teachers are receiving, or how designated resources are used. Asking questions such as these allow policy and decision makers to determine if the coaching program is being implemented as planned and meets the criteria established for a high-quality coaching program.

Measures of implementation, resource use, and identification and handling of unanticipated consequences provide information for program improvement. They are measures of the quality or merit of a coaching program, yet they are not measures of its impact. Another common measure is the perceived value or worth of professional learning. Teachers may appreciate the support they are receiving, yet their appreciation is not evidence that coaching has changed their practice or student achievement. Principals may appreciate coaches and value their contributions to the school staff, yet it may be evident when examining how coaches spend their time, that they spend little time in classrooms supporting or refining implementation of instructional practices.

While effectiveness ¹ measures are essential for managing and improving professional learning, assessing impact is crucial to determine if it is changing instruction or student learning. Assessing the impact of professional learning requires data to measure effectiveness ², whether changes are occurring in both the primary and secondary clients.
Outcomes
Measures of outcome attainment provide evidence of impact, while measure of implementation, resource use, and identification and handling of unanticipated consequences provide information for program improvement. To assess the impact of professional learning, those designing, implementing, and managing professional learning first invest in designing a sufficient, well-resourced, and sustained professional learning program that intends to accomplish specific outcomes. Outcomes describe the expected changes professional learning seeks to achieve for both primary and secondary beneficiaries. They include changes knowledge, attitudes, skills, aspirations, or behaviors (Killion, 2018). When delineated clearly, outcomes shape the professional learning from its content, learning designs including duration, implementation support, and necessary resources.
Primary and secondary beneficiaries
Delineating primary and secondary beneficiaries of professional learning specifies the immediacy of and sequence of change. Primary beneficiaries of professional learning are those educators engaging in learning; secondary beneficiaries are those who are affected by educators' learning. If teachers are participating in professional learning, they are the primary clients; their students are the secondary beneficiaries. If principals are engaging in professional learning, teachers and students are the secondary beneficiaries.
An 8-step process
The evaluation of professional learning follows an eight-step process. Each step, listed below, incorporates a series of decisions to make and processes to follow to conduct a fair, reliable, and thorough evaluation of professional learning.

Assess evaluability to determine whether the staff development program is ready to be evaluated.

Formulate evaluation questions. The questions an evaluation attempts to answer shape the evaluation's design.

Construct evaluation framework to determine the evidence needed to answer the evaluation questions, the data sources, the data collection methodology, logistics of data collection, and the data analysis methods.

Collect data.

Organize, analyze, and display data.

Interpret data to determine merit, worth, and/or impact and to make recommendations for improvement.

Report and use evaluation findings. Identify audiences to receive findings, the most appropriate format for communicating findings to each, and disseminate findings.

Evaluate the evaluation. Reflect on the evaluation process, the knowledge and skills of the evaluation team, the resources and methodologies used, and the findings to improve future evaluations.

Whether large-scale professional learning programs, such as those to implement new curricula or pedagogy, or small-scale professional learning efforts such as those occurring within professional learning communities, evaluation holds stakeholders accountable for results, provides information for making ongoing adjustments, and measures quality and impact of professional learning.
Further reading from Joellen Killion:
8 Smooth Steps, JSD FALL 2003

Why Evaluations Fail, JSD FALL 2017

Assessing Impact: Evaluating Professional Learning, 3rd Edition This third edition guides administrators, professional learning leaders, school improvement teams, and evaluators step by step through the rigors of producing an effective, in-depth, results-based analysis of your professional learning programs.

# data[].attributes.attachment_link.title
Assessing The Impact Of Professional Learning

# data[].attributes.attachment_link.description
To make informed decisions about professional learning, educators use data gathered from formative and summative evaluations.

# data[].attributes.attachment_link.readable_text
When districts invest in professional learning, they expect results for both educators and students. Yet far too often, the evidence of effect is elusive. This reality can be changed when district and school leaders plan for the evaluation of professional learning when they initially design it to strengthen educator effectiveness and student success.

Six districts (Boston Public Schools (Mass.); Greece Central School District (NY); Jenks Public Schools (Okla.); Metro Nashville Public Schools (Tenn.); Prior Lake Savage Area Schools (Minn.); and Shaker Heights City Schools (Oh.) joined the Frontline Research & Learning Institute and Learning Forward to explore how to measure the impact of their professional learning. When planning professional learning and its evaluation, leaders in these districts set out to answer a series of questions to guide both the planning of their professional learning and its evaluation. Each district entered the small-scale, collaborative study with the intention to evaluate an existing or new professional learning initiative and with a desire to learn with and from each other districts engaged in the same work.

The effort of these six districts offers other districts a glimpse of what it takes to design and implement effective professional learning for educators and evaluate its results. While the duration of the study was too short to realize the desired impact of professional learning, it did provide participating districts an opportunity to grapple with the answers to key questions about evaluating professional learning. Learnings emerging from the study are highlighted in this blog.

The quality of an evaluation of professional learning depends on the quality of its planning and implementation.

As the six district leaders weighed the decision to join the study, they first grappled with the question about what to evaluate. Some had existing professional learning programs in place that were evaluable, that is ready to be evaluated (Killion, 2018). Others had professional learning programs that were not yet evaluable and required more clarity in terms of outcomes for educators and students, indicators of success, and measures of success. Others were in the early phases of developing professional learning programs to meet identified district needs.

To evaluate the implementation or effectiveness both on educators and students, evaluation requires clear delineation of desired outcomes, indicators of success, and viable measures of the outcomes. This means that district or school leaders shift the focus of professional learning evaluation to assessing coherent programs rather than isolated or occasional events.

Using extant data sources expedites evaluation of professional learning yet may limit interpretations.

As district leaders launched comprehensive professional learning programs, as Boston Public Schools did, they wanted to know if their efforts were having the desired effects. To measure their efforts, Boston leaders looked to the multiple data sources already existing in the district to consider which might be useful as indicators of the impact of their new professional learning program.

As do most districts, Boston leaders had multiple sources of data that could be repurposed without creating a data burden (Killion, 2018). However, since the data sources already in place are approximations of measures of the specific outcomes of the professional learning initiative, district leaders analyzed and interpreted the results to form conclusions about the professional learning program with thoughtful consideration regarding the data from approximate measures.

A high-quality professional learning system meets the needs of district educators and increases coherence and effectiveness of professional learning programs.

When professional learning within a district or school lacks coherence, it is difficult to measure its effects. In Jenks Public Schools, participation in the study meant an opportunity to solidify the district professional learning system and to create a planning process that ensured that professional learning met the criteria of quality from its inception.

Using a planning model, the district and school leaders are able to align professional learning with identified needs, provided adequate implementation support, and monitor implementation to increase the likelihood of results. This system includes a decision-making model for planning and implementing high-quality professional learning. This means that the professional learning educators experience is specifically designed to produce results both for them and for their students.

Existing and new programs both require evaluation, often for different purposes.

Whether professional learning programs are new in a district or long-standing, evaluation is necessary. Existing programs, such as the mentoring and induction program offered in Greece Central School District through its Teacher Center, or a new coaching program in Shaker Heights City Schools, evaluation provides valuable information for multiple purposes. In Greece Central School District, results of the evaluation inform decisions for refining the existing programs to be more beneficial to participants. Annual data collection from multiple sources about the program inform the continuous upgrades to the program. Evaluation in this case is for the purpose of refining an existing program.

In Shaker Heights City School District, evaluation offers data about the new coaching program for two purposes. First, the data collected guides coaches' work with teachers. As the coaching program in Shaker Heights matures, the data collected can help district leaders measure effects of the program and make necessary adjustments to strengthen the program. In both districts, evaluation serves different purposes. Evaluation in this case is for program management and implementation.

District leaders in Prior Lake Savage Area Schools wanted to implement a districtwide instructional framework to support student learning. They wanted to engage multiple stakeholders in the design process so that staff both had a voice in the development of the framework as well as gained an awareness of the need for the framework. Using an innovative data gathering process, the district leadership team collected data from teachers, principals, and central office staff and analyzed the data collected for common themes. By engaging staff, the leadership team built a common understanding of the need for and a shared purpose for the instructional framework. These early data provide baseline data that can be used for pre- and post-implementation evaluation. Evaluation in this case is for program planning and for collecting baseline data that may be useful in a pre- post evaluation design.

In Metro Nashville schools, the Collaborative Inquiry Process, a system for collecting, analyzing and using a variety of data to improve schools, was developed through an alliance between REL Appalachia and the district. After the design and pilot testing phase, district leaders were ready to implement the process in five district schools, since the unit of change is not a school but rather a cluster of individuals within a school. This is the case with the Collaborative Inquiry program designed in cooperation with the Appalachian Regional Laboratory and available to teams of teachers within Metro Nashville schools. The expected unit of change in a professional learning program both influences the program's design and implementation as well as its evaluation. Evaluation in Metro Nashville focused on measuring the quality of implementation and impact on each school's student achievement.

Sound, flexible data systems make the evaluation more efficient and effective.

Useful evaluations take time, resources and effort. Districts with data systems that allow them to gather, track, analyze and access data quickly are able to use data frequently to monitor the professional learning program's and participants' progress and inform decisions to adjust their programs.

Data systems that generated analytics using multiple types of educator and student data allowed district leaders to adapt the data collected to measure both levels of effectiveness, first on educators and secondly on students. Sound data systems shifted district leaders' effort from collecting data to using data to make informed decisions about professional learning and its impact by allowing them to analyze, interpret and use data on professional learning more rapidly to strengthen programs and impact on participants and their students.

As district leaders continue to explore effective and efficient ways to evaluate the effectiveness of their professional learning programs, they will realize the benefits of evaluation for program planning, monitoring implementation, and measuring impact on both educators and students. If professional learning is to impact educator effectiveness and student results, district leaders must commit to ongoing evaluation to inform all decisions about professional learning.

Killion, J. (2018). Assessing impact: Evaluating professional learning, 3rd edition. Thousand Oaks, CA: Corwin.

# data[].attributes.attachment_link.title
Measuring the Impact of Professional Learning

# data[].attributes.attachment_link.description
With Frontlines Professional Learning (designed to increase educator effectiveness and student success), you can expect results for educators and students.