Quant - comparative between groups data on cohort attendance, VLE engagement, assessment submission, assessment results data, Panopto data etc.

Qual - qualitative anonymous survey co-designed with students; student focus groups for deeper dive; colleague focus groups

Working with professional services and digital learning colleagues re. data

Measuring engagement

1. Collect Staff Feedback on Curriculum Design Roadshows:
We will gather detailed feedback from academic staff who participate in our Curriculum Design roadshows at Sunderland, London, and Hong Kong campuses. This will involve post-session surveys, structured interviews, and focus groups to understand their confidence in applying the new curriculum design principles. Key evaluation metrics will include their perceived relevance, usability, and how well the roadshows equipped them to use Jisc's curriculum design model in their work.

2. Assess Student Voice Feedback:
The student voice will be crucial in measuring the effectiveness of curriculum changes. We will incorporate targeted questions into our annual digital student experience survey, aiming to capture students' perceptions of the newly implemented curriculum design principles. We will also engage student representatives through the Enhancement to the Student Experience Group (ESEG), gathering insights on how the curriculum has influenced their learning experiences and overall satisfaction.

3. Measure the Adoption of Curriculum Design Principles:
A significant outcome will be the visible integration of Jisc’s curriculum design process model into our institutional practices. We will evaluate how confidently and comprehensively academic staff apply these principles by conducting audits of program and module design documentation. These audits will check for consistent use of the new principles and will assess how well these have been embedded into course planning and delivery.

4. Track Institutional Approval and Integration:
We will monitor progress through the stages of institutional approval, capturing meeting minutes and feedback from key committees, such as ESEG and the Student Success Committee (SSC). Evidence of formal adoption and endorsement will serve as a key measure of success, alongside any policy changes reflected in our quality handbook and programme validation processes.

5. Conduct Pre- and Post-Pilot Comparative Analysis:
We will perform a comparative analysis between baseline data collected prior to the pilot and post-pilot data. This will include metrics such as staff engagement levels, changes in curriculum quality as assessed by Threshold Standards audits, and improvements in program design reflected in audit reports.

6. Evaluate Changes in TEF and Quality Assurance Contributions:
Our evaluation will look at how well the pilot outcomes contribute to our Teaching Excellence Framework (TEF) submission. We will document specific improvements linked to the curriculum design model and how these are evidenced in program validation and quality assurance processes.

7. Longitudinal Monitoring and Reporting:
Finally, we will implement a longitudinal evaluation to track the sustainability of changes over time. We will continue to monitor student experience and satisfaction metrics annually and report on the impact at key academic conferences, including our University of Sunderland Learning and Teaching Conference.

In summary, our comprehensive evaluation framework will ensure that we capture the impact of the pilot from multiple angles, integrating staff feedback, student voice, and institutional data to measure the success of the curriculum design transformation.

Engagement metrics

Student feedback

NSS outcomes around resources, VLE, learning environments, intellectually stimulating and engaging

Reducing the awarding gap

‘You said, we did’ as both communication and evaluation

Recognition by external examiners

How to change culture (to embrace beyond blended) amongst staff who support learning and teaching.

Where to start in the strategic landscape?!

•JISC qualitative anonymous survey co-designed with Level 6 student researchers.

•Thematic analysis will inform…

Stage 2:

•L6 Student-researcher and staff led follow-up focus group interviews with participants

•Thematic analysis feedback into module design

Stage 3:

•Between groups comparative study AY23.24 and AY24.25 on cohort attendance, VLE engagement, assessment submission, assessment results data

•Independent groups parametric T-Tests (normal distribution); independent groups non-parametric T-tests (non-normal distribution)

Stage 4:

•Staff focus groups

•Thematic analysis feedback into Pre-Live-Post model

Currently going through ethics to turn this into a research project, but key sticking point is consent to use previous cohort data. Any ideas gratefully received :) Alison

baseline the students confidence, expectations of what their responsibilities for engagement

Look at VLE data conduct focus groups

Logos, colour and branding can also be vocabulary

Different groups of stakeholders may need some translation as they will use different languages.

I could use the comparing live and asynchronous time and comparing in-place and online sessions resources as a basis for group activities around sharing practice and thinking about different approaches to teaching.

I could do the same with the session types – but here maybe make more direct links to pedagogical theories, and overall development of critical perspectives on modes of teaching and learning."

It would be interesting to hear more ideas around this envisaged usage.