# data[].attributes.body
Includes a digital glossary of key terms in HE evidence use

# data[].attributes.body
Sign up for a free account for this evaluation planning tool

# data[].attributes.attachment_link.readable_text
Welcome to the Universal Evaluation Framework (UEF).

This is an open access tool for planning evaluations or for gathering evaluative evidence created by Prof Stella Jones-Devitt and Prof Liz Austen.

Please click on the link below to create your individual account, or sign-in via the Login prompt if you already have an account. You will receive an email confirmation when your request for an account has been approved. If you haven't received this confirmation in 3 working days please contact scolpp@staffs.ac.uk

Any data you provide for registration will only be used to create this account and will not be used for any other purpose.

Once you have access, please read the User Guide which will provide a comprehensive overview of the UEF and suggestions for use. We hope you have fun using this tool and it leads to crucial learning of effectiveness and for enhancement!

Liz and Stella

Signup for an account

Copyright 2024 Evaluation for All.

# data[].attributes.attachment_link.description
Welcome to the Universal Evaluation Framework (UEF). This is an open access tool for planning evaluations or for gathering evaluative evidence created by Prof Stella Jones-Devitt and Prof Liz Austen. Please click on the link below to create your individual account, or sign-in via the Login prompt if you already have an account.

# data[].attributes.body
Has a great search function of evaluation key concepts

# data[].attributes.attachment_link.description
BetterEvaluation is a knowledge platform and global community. Our vision is a world of better evaluation, better decisions, and better results for people and the planet.

# data[].attributes.attachment_caption
This is a cross-sector group of like-minded evaluation advocates working to enhance higher education student outcomes.

# data[].attributes.body
There's some great Theory of Change resources on the TASO website Theory of Change - TASO

# data[].attributes.attachment_link.title
Resources | methods@manchester | The University of Manchester

# data[].attributes.attachment_link.description
Watch video recordings of our latest sessions, seminars and lectures.

# data[].attributes.attachment_link.readable_text
Higher education institutions are constantly being told to make higher education better for students, and these demands are steeped in political rhetoric.

In the world of vacuous policy promises and accompanying public indifference, we offer some spin-free enhancement points for HE policymakers to action, in getting them productively across the finish line for student success. Sounds easy: so why does achieving the holy grail of student success appear so problematic? In this blog post, we imagine a world in which key decision-makers boldly go where they very often fear to tread. Into explicit evidence-informed effective action for enhancement.

We suggest that senior policymakers with responsibility for learning and teaching enhancement at organisational and sectoral level need to have more explicit and strategic ownership and accountability for envisioning what great evaluation looks like, rather than being exasperated when such enhancements don't cohere or provide desired outcomes.

Drawn from evidence gleaned from our near decade-long work as HE evaluators, such as this longitudinal Evaluation of 20 Years of Enhancement Themes for QAA Scotland published earlier this year, we offer five key promises that these strategic policymakers need to make, alongside those in wider HE funding roles who might benefit from adopting some of these promises, too.

The promises are translated into a set of "we will" pledges for policymakers to adopt, if they truly want to make a positive difference for, and with, students. We ask you to consider why "we will" invariably becomes "we won't" or "we can't."

Pledge 1: We will align all enhancement work within an overarching organisational framework.

Within higher education institutions, maximum effectiveness will be achieved by aligning all enhancement-led projects and activities within an overarching framework which will demonstrate clear assessment of contribution made to strategic ambitions. Activities will be rated against a set of shared and transparent outcomes. The alternative is an activity focus rather than an outcome focus - well-meaning but lacking an evaluative mindset.

Pledge 2: We will ensure there is a well-resourced "critical mass" in place to drive enhancement work and organisational success.

All enhancement work will be aligned with strategic change to ensure that a critical mass for momentum is established. Ensuring strength in numbers within an institution will drive effective resourcing, better engagement, understanding and, most importantly, impact. This avoids silos and lone early-adopters.

Pledge 3: We will ensure that enhancement work is integrated into a change process for maximum impact.

Enhancement work is more effective when planned, monitored, evaluated and implemented within a strategic change process that can make a difference. We will consciously involve a range of stakeholders in any decision-making to routinely reflect on impact and any future revisions, scaling or stopping.

Pledge 4: We will focus on impact, not number of outputs

Rather than only encouraging the production of a plethora of outputs (toolkits, practice guides, reports, case studies etc), we will ensure consideration of how activity drives benefit and impact, especially concerning the contribution made to achieving positive student outcomes. The production of a hundred plus outputs doesn't necessarily add up to impact.

Pledge 5: We will prioritise improving student outcomes by having clear indicators of success from the start.

All enhancement work will include projection of impact on a range of student outcomes. Success will be considered as a long-term venture relative to the planned activity. Long-term student outcomes relating to improving the metrics on continuation, attainment, completion and progression will be a priority when institutions develop strategic enhancement approaches. Intermediate outcomes are our friends - they can ensure continued enhancement funding whilst we wait for the evidence of assumed longer term impact.

So, there you have it. Five key evidence-informed policymaker promises concerning enhancement work that should make a real difference to achieving positive student outcomes. Simple, right? Unfortunately, our near decade of evaluation experience, along with a burgeoning evidence-base, indicates that very few enhancement approaches go beyond planning enhancement work premised on emotional desirability. This inevitably involves lots of well-meaning activity and outputs, in which everyone participating has a jolly time, yet characterised by a lack of focus concerning the difference any of this makes for improving student outcomes.

Changes to access and participation plans (in England), TEF and the emerging Tertiary Framework in Scotland are more explicit than ever about the connections between institutional activities and impact on student outcomes. These pledges therefore become evaluative reminders for effective policy enactment. How can your institutional policymakers move from a "we won't" or"'we can't" model when engaging in enhancement work for student success?

# data[].attributes.attachment_link.title
A challenge for senior leaders to act on evaluation evidence for improving student outcomes

# data[].attributes.attachment_link.description
Evaluation really does show what works when it comes to students' outcomes, but that doesn't mean that's what happens. Liz Austen and Stella Jones-Devitt call on policymakers to take the pledge

# data[].attributes.headline
Musing about equality of opportunity regulation

# data[].attributes.subject
Musing about equality of opportunity regulation

# data[].attributes.body
Check out the honorary name check for Liz!

# data[].attributes.attachment_link.title
What’s next in equality of opportunity regulation - Office for Students

# data[].attributes.attachment_link.description
Read John Blake’s speech at the Universities UK Access, Participation and Student Success conference, where he reflects on the current round of Access & Participation Plans and the next stage of equality regulation in English higher education.

# data[].attributes.body
Evaluating and Researching University Participation Interventions

https://www.nerupi.co.uk/

# data[].attributes.headline
The Everyday Change Play Book: Because change doesn't need to be shit

# data[].attributes.subject
The Everyday Change Play Book: Because change doesn't need to be shit

# data[].attributes.body
A book about evaluative thinking which is an easy and enjoyable read!

# data[].attributes.attachment_link.authors[].name
Susie Palmer-Trew (Author), Jan Jayes (Author)

# data[].attributes.attachment_link.title
The Everyday Change Play Book: Because change doesn’t need to be shit.

# data[].attributes.attachment_link.description
Buy The Everyday Change Play Book: Because change doesn’t need to be shit. by Palmer-Trew, Susie, Jayes, Jan (ISBN: 9798362175948) from Amazon’s Book Store. Everyday low prices and free delivery on eligible orders.

# data[].attributes.body
Another short and to the point book on evaluation by the evaluation icon David Parsons

# data[].attributes.attachment_link.description
Demystifying Evaluation - Practical Approaches for Researchers and Users; Demystifying evaluation is an accessible introductory guide explaining the options open to evaluators and how to make appropriate choices of research methods and covering issues such as managing expectations of evaluation, quantitative and qualitative methods, engaging stakeholders and providing action-orientated approaches to help end-users.

# data[].attributes.body
A plug for embedding co-creation and Student Partnership approaches into our evaluations.

The link above is to a co-created project we did for the last Enhancement Theme (in Scotland) - scroll to the 'Student Champions' tab to see a student produced video and output.

We worked with Student Champions to articulate and map Learner Journeys, and they were great at devising novel ways of gathering evidence and input from other students (and were fervently against doing 'another survey')

They devised and led engagement drives in the canteen (with sweets and dogs!), QR codes to Padlets, and student led focus groups. (Feel free to get in touch if you'd like any additional info via Catriona.E.Bell@gmail.com)

# data[].attributes.attachment_link.title
Enhancement Themes Project Outputs | LEAD | Queen Margaret University

# data[].attributes.attachment_link.description
This webpage provides a list of the outputs from work undertaken through the QMU institutional project for the Resilient Learning Communities (RLC) Enhancement Theme that ran between 2020 and 2024.

# data[].attributes.body
Hoping to find inspiration here :)

# data[].attributes.attachment_link.title
Impact evaluation with small cohorts (evaluation guidance) - TASO

# data[].attributes.attachment_link.description
Impact evaluation is important but can be challenging. TASO promotes the use of rigorous experimental and quasi-experimental impact evaluation methodologies as these are often the best way to determine causal inference. However, sometimes these types of impact evaluation raise challenges: They are effective in describing a causal link between an intervention and an outcome, but […]

# data[].attributes.body
Remember Chris (aka Liz)? Here are some possible responses to their problem, in the format of a snazzy zine

# data[].attributes.body
ChangeBusters sessions can build confidence and capacity in evaluation

# data[].attributes.attachment_link.title
Exploring Theory of Change using ChangeBusters | Advance HE

# data[].attributes.attachment_link.description
Support your staff to understand Theory of Change with a ChangeBusters workshop delivered at your institution.

# data[].attributes.attachment_link.readable_text
Small n impact evaluations are evaluation designs that involve only a small number of cases, insufficient to construct a 'traditional' counterfactual impact evaluation in which statistical tests of the difference between an intervention group and a control group are used to estimate impact. Instead of a counterfactual design, mid-level theory - explaining why the intervention will have an impact - together with alternative causal hypotheses are used to establish causation beyond reasonable doubt.

Small n impact evaluation designs should be considered when one or more of the following are true:

There is one case or a small number of cases. A case could be a person, an organisation, a school or a classroom.

There is no option to create a counterfactual or control group.

There is considerable heterogeneity in the population receiving the intervention, the wider context of the intervention or the intervention itself. Such heterogeneity would make it impossible to estimate the average treatment effect using a traditional counterfactual evaluation, as different sub-groups will be too small for statistical analysis (White and Phillips 2012).

There is substantial complexity in the programme being evaluated, meaning that an evaluation designed to answer the question 'does the programme cause outcome X' may make little sense, whereas an evaluation design that recognises that the programme is just one ingredient in a 'causal cake' may make more sense.

In these situations, a small n impact evaluation still offers the possibility of making causal statements about the relationship between an intervention and an outcome.

Important limitations of small n methods include:

Many of these methods do not allow evaluators to quantify the size of an impact. This can also reduce options for subsequent economic evaluation.

These methods all involve detailed information gathering at the level of the case; in some small n impact evaluations, gathering in-depth qualitative data from one or more cases could be just as time-consuming and resource-intensive as data collection in a traditional, counterfactual impact evaluation.

All evaluation requires that evaluators have the necessary skills in evaluation design, data collection, analysis and report writing. However, in a small n impact evaluation, the evaluator additionally needs a deeper knowledge of the programme and the context within which it is being implemented than might typically be required in a traditional, counterfactual impact evaluation.

# data[].attributes.attachment_link.description
Small n impact evaluations are evaluation designs that involve only a small number of cases, insufficient to construct a ‘traditional’ counterfactual impact evaluation in which statistical tests of the difference between an intervention group and a control group are used to estimate impact. Instead of a counterfactual design, mid-level theory – explaining why the intervention […]

# data[].attributes.headline
Evidence for Enhancement (from previous Enhancement Theme in Scotland)

# data[].attributes.subject
Evidence for Enhancement (from previous Enhancement Theme in Scotland)

# data[].attributes.attachment_link.readable_text
This report evaluates the impact of the 2017-20 Enhancement Theme - Evidence for Enhancement: Improving the Student Experience.

Author:

QAA Scotland

Format:

PDF

Size:

2.64 MB

Author:

QAA Scotland

Format:

PDF

Size:

2.87 MB

# data[].attributes.attachment_link.description
Evidence for Enhancement: Improving the Student Experience is the current Enhancement Theme and will run between 2017-2020. The Scottish sector engage with the Theme in a variety of ways including institutional activity, collaborative clusters and sector-wide activity.

# data[].attributes.body
Maybe a useful resource for those evaluating CPD e.g. Fellowship, Conferences, taught courses etc. Kneale, P., Winter, J., Turner, R., Spowart, L., Hughes, J. and Muneer, R. (2016) Evaluating Teaching Development in Higher Education – A toolkit. Advance HE. [Online] Available at: https://s3.eu-west-2.amazonaws.com/assets.creode.advancehe-document-manager/documents/hea/private/evaluating_teaching_development_in_he_-_toolkit1_1568037347.pdf [accessed 04 February 2024].