# data[].attributes.attachment_link.readable_text
TECH6102 E-PORTFOLIO

Dear visitors, It is my pleasure to have you here and share my experience as a student in the Instructional and Learning Technology (ILT) Department in Squ. This is where you can find all my work in the course of Evaluating & Assuring Quality for Alternative Learning Environment

TECH6102.Thank you for visiting my e-portfolio, and I hope you find it informative and inspiring. Enjoy your exploration!​

To play, press and hold the enter key. To stop, release the enter key.

# data[].attributes.attachment_link.description
TECH6102 E-PORTFOLIO Dear visitors, It is my pleasure to have you here and share my experience as a student in the Instructional and Learning Technology (ILT) Department in Squ. This is where you can find all my work in the course of Evaluating & Assuring Quality for Alternative Learning Environment TECH6102.Thank you for visiting my e-portfolio, and I hope you find it informative and inspiring.

# data[].attributes.attachment_link.description
Hi, I’m Salma Al Barwani, an English Language Instructor from Muscat | Oman.

# data[].attributes.attachment_link.description
Not your computer? Use Private Browsing windows to sign in. Learn more about using Guest mode

# data[].attributes.attachment_link.readable_text
This course is part of the Master's curriculum in Instructional and Learning Technologies. It aims to provide a working understanding of the evaluation and quality assurance models in alternative learning environments such as e-learning, distance learning, and hybrid alternatives. It specifically aims to analyze results to improve the performance of students and the effectiveness of e-learning programs emanating from vendors and providers.

To play, press and hold the enter key. To stop, release the enter key.

# data[].attributes.attachment_link.description
This course is part of the Master's curriculum in Instructional and Learning Technologies. It aims to provide a working understanding of the evaluation and quality assurance models in alternative learning environments such as e-learning, distance learning, and hybrid alternatives.

# data[].attributes.attachment_link.description
Not your computer? Use Private Browsing windows to sign in. Learn more about using Guest mode

# data[].attributes.attachment_link.description
About me: IT teacher with more than 5 years of experience instructing students from grade 1 to grade 9 in Muscat. Received many appreciation certificates every year for excellent performance in all schools worked in. Trained many teachers and collaborated with charity organizations to train

# data[].attributes.body
Please upload the link to your portfolio here.

# data[].attributes.headline
Standards and competencies for IDT professionals

# data[].attributes.subject
Standards and competencies for IDT professionals

# data[].attributes.attachment_link.readable_text
Students entering the field of instructional design must possess a wide array of competencies to be successful in their future roles (Ritzhaupt & Martin, 2014). Competencies are the knowledge, skills, and abilities professionals need in their roles, while standards speak to a pre-defined level of quality or attainment of those competencies. Competencies and standards are essential aspects to advance professionals in this field. Several professional organizations guide the development of competencies and standards. They also have certification programs for instructional designers and instructional programs. In this chapter, we review the instructional design standards and competencies both from professional organizations and those proposed by researchers who guide the educational preparation of instructional designers and also support their academic and work experiences.

Competency and Standard

In this section, we review the term competency and standard before we introduce instructional design competencies from professional organizations and from research. Richey et al. (2001) defined competency as "a knowledge, skill or [ability] that enables one to effectively perform the activities of a given occupation or function to the standards expected in employment" (p. 26). Spector and De la Teja (2001, p. 2) refer to the term competency as "a state of being well qualified to perform an activity, task or job function" and competency refers to the "way that a state of competence can be demonstrated to the relevant community." Thus, competencies are specific to a community of endeavor in which professionals determine the competencies valuable to the profession. As competencies are identified and developed, professionals express these competencies as standards to assist professionals, professional associations, academic programs, and the larger community to better understand the domain of interest.

The KSA framework, comprised of Knowledge, Skills, and Abilities, has been used by researchers to study competencies in the field. Ritzhaupt et al. (2010) used the KSA framework to categorize educational technology competencies into knowledge, skills and abilities statements. Figure 1 illustrates this framework in light of three domains used to characterize the field: creating, using, and managing. The KSAs represent the core processes and resources used by those practicing in the field, which are the creation of instructional materials, learning environments, and instructional products using systematic approaches and based on research to improve learning and performance. Using refers to selecting, using, and implementing educational technologies and processes to support student learning and to enhance their pedagogy. Management refers to managing people, processes, physical infrastructures, and financial resources to create diverse learning environments and provide supportive learning communities to improve learning and performance (AECT Standards 2012, 2008).
Figure 1 Knowledge, Skill, and Abilities Statements for Educational Technologists (Adapted from Ritzhaupt et al., 2010)
Standards are critically important to establish a foundation for a field. For instance, the field of project management established the well-known American National Standards Institute's (ANSI) Guide to The Project Management Body of Knowledge (PMBOK), which is used as the basis for the Project Management Professional (PMP) certification program and as the official body of knowledge for the profession.

Instructional Design and Technology Competencies and Standards From Professional Organizations

The field of instructional design is comprised of several professional organizations, several of which define competencies and standards for the profession. Table 1 provides a summary of these professional organizations and the following section provides more details about each. Each organization has a different focus and provides standards and competencies for their relevant programs. Students should be reminded that these standards and competencies serve as ideal frameworks, and should not be discouraged by their scope.
Table 1 Professional Organizations Who Publish Instructional Design and Technology Standards
International Board of Standards for Training, Performance and Instruction (IBSTPI)
http://ibstpi.org/
Ibstpi Vision: To be the leader in setting international standards in the areas of training, instruction, learning, and performance improvement.

Ibstpi Mission: Develop, validate, and promote implementation of international standards to advance training, instruction, learning, and performance improvement for individuals and organizations.

Ibstpi has competency sets for various learning and development roles, including the instructional designer. They also have competency sets for other roles such as training manager, evaluator, instructor, and learner. For the instructional designer, Ibstpi (2012) developed 22 competencies across five domains.

1. Professional Foundations

2. Planning and Analysis

3. Design and Development

4. Evaluation and Implementation

5. Management

Each of these competencies has detailed performance statements and a level of expertise (essential, managerial and advanced) identified for each of them. Ibstpi goes through a rigorous development model to identify and validate these competencies. The steps in the model include preliminary analysis of job roles, identification of foundational research, competency drafting by directors and experts, validation study design, translation of research instruments in multiple languages and implementation worldwide with working professionals, data analysis and competency validation, publishing final competencies and performance statements and disseminating the competencies to practitioners, researchers and organizations.

International Society for Technology in Education (ISTE)
https://www.iste.org/
ISTE Vision: ISTE's vision is that all educators are empowered to harness technology to accelerate innovation in teaching and learning, and inspire learners to reach their greatest potential.

ISTE Mission: ISTE inspires educators worldwide to use technology to innovate teaching and learning, accelerate good practice, and solve tough problems in education by providing community, knowledge, and the ISTE Standards-a framework for rethinking education and empowering learners.

ISTE has developed well-adopted standards for students, teachers, administrators, coaches, and computer science educators. The ISTE standards are widely accepted in the K-12 community, and have been transformed into assessment systems (Hohlfeld et al., 2010) and a new professional credential offered by ISTE known as the ISTE Certification, which is a vendor neutral teacher certification based on the ISTE Standards for Educators. The ISTE Standards for Educators can be accessed at https://www.iste.org > standards for more information.

They include:

1. Learner: Educators continually improve their practice by learning from and with others and exploring proven and promising practices that leverage technology to improve student learning.

2. Leader: Educators seek out opportunities for leadership to support student empowerment and success and to improve teaching and learning.

3. Citizen: Educators inspire students to positively contribute to and responsibly participate in the digital world.

4. Collaborator: Educators dedicate time to collaborate with both colleagues and students to improve practice, discover and share resources and ideas, and solve problems.

5. Designer: Educators design authentic, learner-driven activities and environments that recognize and accommodate learner variability.

6. Facilitator: Educators facilitate learning with technology to support student achievement of the ISTE Standards for Students.

7. Analyst: Educators understand and use data to drive their instruction and support students in achieving their learning goals.

International Society for Performance Improvement (ISPI)
https://www.ispi.org/
ISPI Vision: Performance improvement practices are recognized globally as an essential part of every organization's competitive strategy.

ISPI Mission: ISPI and its members use evidence-based performance improvement research and practices to effect sustainable, measurable results and add value to stakeholders in the private, public, and social sectors.

ISPI has proposed 10 Human Performance Practitioner Standards for instructional designers who assume the specialized role of performance consultants. The ten standards include,

1. Focus on Results or Outcomes

2. Take a Systemic View

3. Add Value

4. Work in Partnership with Clients and Stakeholders

5. Determine Need or Opportunity

6. Determine Cause

7. Design Solutions including Implementation and Evaluation

8. Ensure Solutions' Conformity and Feasibility

9. Implement Solutions

10. Evaluate Results and Impact

In addition to the practitioner standards, ISPI also has accreditation standards for organizations and programs/courses. ISPI certifies practitioners though a rigorous peer-review process and with the opportunity for the practitioners to be re-certified every three years.

Association for Talent Development (ATD)
https://www.td.org/
ATD Vision: Create a World That Works Better

ATD Mission: Empower Professionals to Develop Talent in the Workplace

ATD certifies professionals in learning and performance (CPLP) and associate professionals in talent development. The Certified Professional in Learning and Performance (CPLP) candidates are tested on ten (10) areas of expertise and include:

1. Performance Improvement

2. Instructional Design

3. Training Delivery

4. Learning Technologies

5. Evaluating Learning Impact

6. Managing Learning Programs

7. Integrated Talent Management

8. Coaching

9. Knowledge Management

10. Change Management

ATD also has a competency model for learning and development through which they identify roles, areas of expertise, and foundational competencies for professionals in learning and performance.

Association for Educational Communications and Technology (AECT)
https://www.aect.org/
AECT Vision: We seek to be the premier international organization in educational technology, the organization to which others refer for research and best practices.

AECT Mission: Provide international leadership by promoting scholarship and best practices in the creation, use, and management of technologies for effective teaching and learning.

Januszewski and Molenda (2007) defined Educational Technology as "the study and ethical practice of facilitating learning and improving performance by creating, using, and managing appropriate technological processes and resources" (p.1).

AECT has developed standards for educational technologists in five areas. These standards can be accessed from the AECT website.

1. Content Knowledge

2. Content Pedagogy

3. Learning Environments

4. Professional Knowledge and Skills

5. Research

For each of the standards, there are several indicators provided. AECT certifies graduate certificate programs in higher education who prepare educational technologists based on these standards.

University Professional and Continuing Education Association (UPCEA)
https://upcea.edu/
UPCEA is a leading association of professional, continuing and online education. Their goal is to provide high quality, professional, continuing and online education programs of practice in higher education.

UPCEA® Purposes:

To promote quality in professional and continuing higher education.

UPCEA has seven standards identified to provide excellence in online learning leadership.

1. Internal Advocacy

2. Entrepreneurial Initiative

3. Faculty Support

4. Student Support

5. Digital Technology

6. External Advocacy

7. Professionalism

Online Learning Consortium (OLC)
https://onlinelearningconsortium.org/
OLC® Vision: Setting the global standard in online and digital learning

OLC® Mission: Creating community and connections around quality online and digital learning while driving innovation

OLC's Five Pillars of Quality Online Education include:

1. Learning Effectiveness

2. Scale

3. Access

4. Faculty Satisfaction

5. Student Satisfaction

Instructional Design and Technology Competencies From Research

In addition to the professional organizations, several researchers have examined instructional design competencies and standards over the years. Table 2 below provides details of researchers and the competencies and standards examined for various instructional design professionals. These articles can be used to plan professional development, academic programs, and learning experiences for our professionals and emerging professionals.
Table 2 Instructional Design and Technology Competencies From Research

Liu, Gibby, Quiros, and Demps (2002)

Instructional Designers

Interviews

Problem-solving and decision-making skills

Kenny, Zhang, Schwier and Campbell (2007)

Instructional Designers

Literature Review

Communication skills, knowledge of instructional design models, problem-solving/decision-making skills, and technology skills

Ritzhaupt, Martin and Daniels (2010)

Educational Technologists

Job Announcement Analysis and Survey of Professionals

Multimedia competencies for educational technologists

Lowenthal, Wilson and Dunlap (2010)

Instructional Designers

Job Announcement Analysis

Instructional design experience, communication skills and collaboration skills

Wakefield, Warren and Mills (2012)

Instructional Designers

Job Announcement Analysis

Communication and interpersonal skills, managing multiple instructional

Design projects, specific traits, and collaborative skills

Ritzhaupt and Kumar (2015)

Instructional Designers in Higher Education

In-depth Interviews

Solid foundation in instructional design and learning theory, possess soft skills and technical skills, and have a willingness to learn on the job

Learning theory also guides ethical decision-making when engaged in the creation of a wide-array of learning solutions. Professionals must also stay abreast in emerging learning technologies and should possess both the ability to learn independently and the commitment to lifelong learning. Other knowledge, skills, and abilities were identified in these studies, but these areas noted were frequently observed and noted.

Conclusion

Professional competencies and standards are helpful ways to communicate the value-add of our professionals to stakeholders outside of our community in various professional contexts (e.g., healthcare), to assist our professionals and emerging professionals in planning professional development and lifelong learning (e.g., which webinar to attend), and to guide our academic programs to align with the expectations of the needs in our field (e.g., selecting which topics to cover in an instructional design course). While no list of competencies and standards is complete, those enumerated in this chapter provide readers a glimpse of the status of the profession as described by our professional organizations and existing research literature. Students entering the profession should spend time on learning these competencies and standards to identify career paths and professional development opportunities. We conclude the chapter with some independent learning activities for your edification.

Application Exercises

1. How should professional competencies and standards be identified, documented, and used by professionals in our field? What forms of research methods have been used to identify and document these competencies and standards? Write a brief overview of how you think competencies and standards should be developed in our profession by reviewing the existing articles listed in Table 2.

2. Read three of the recent articles listed in Table 2. Using the competencies and standards provided in these articles, write a short list of professional learning outcomes for yourself to achieve in the next calendar year.

3. Explore one of the professional organizations discussed in this chapter to identify more detailed information about the organization, including when the professional organization hosts its annual conference, the cost of membership, the list of readings available with membership, and any of professional learning (e.g., webinars) provided by the organization for its members.

4. Some scholars, such as Ritzhaupt and Martin (2010; 2014; 2018) have expressed the competencies of professionals using knowledge, skill, and ability statements. Using this approach, search and identify 10 instructional design professional position announcements using tools like indeed.com. After identifying the announcements, code the knowledge, skill, and ability statements found in these announcements.

References

Brown, A., Sugar, B. & Daniels, L. (2007). Media production curriculum and competencies: Identifying entry-level multimedia production competencies and skills of instructional design and technology professionals: Results from a biennial survey. Paper presented at Association of Educational Communcations and Technology.

Hohlfeld, T. N., Ritzhaupt, A. D., & Barron, A. E. (2010). Development and validation of the Student Tool for Technology Literacy (ST2L). Journal of Research on Technology in Education, 42(4), 361-389.

Januszewski, A., Molenda, M., & Harris, P. (Eds.). (2008). Educational technology: A definition with commentary (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associate

Kang, Y., & Ritzhaupt, A. D. (2015). A job announcement analysis of educational technology professional positions: Knowledge, skills, and abilities. Journal of Educational Technology Systems, 43(3), 231-256.

Kenny, R.F., Zhang, Z., Schwier, R.A., & Campbell, K. (2008). A review of what instructional designers do: Questions answered and questions not asked. Canadian Journal of Learning and Technology, 31(1).

Liu, M., Gibby, S., Quiros, O. & Demps, E. (2002). Challenges of being an instructional designer for new media development: A view from the practitioners. Journal of Educational Multimedia and Hypermedia, 11(3), 195-219.

Lowenthal, P., Wilson, B. G., & Dunlap, J. C. (2010). An analysis of what instructional designers need to know and be able to do to get a job. Presented at the annual meeting of the Association for Educational Communications and Technology. Anaheim, CA.

Ritzhaupt, A. D., & Kumar, S. (2015). Knowledge and skills needed by instructional designers in higher education. Performance Improvement Quarterly, 28(3), 51-69.

Ritzhaupt, A. D., & Martin, F. (2014). Development and validation of the educational technologist multimedia competency survey. Educational Technology Research and Development, 62(1), 13-33.

Ritzhaupt, A., Martin, F., & Daniels, K. (2010). Multimedia competencies for an educational technologist: A survey of professionals and job announcement analysis. Journal of Educational Multimedia and Hypermedia, 19(4), 421-449.

Ritzhaupt, A. D., Martin, F., Pastore, R., & Kang, Y. (2018). Development and validation of the educational technologist competencies survey (ETCS): Knowledge, skills, and abilities. Journal of Computing in Higher Education, 30(1), 3-33.

Spector, J. M., & De la Teja, I. (2001). Competencies for online teaching. ERIC Clearinghouse on Information & Technology, Syracuse University.

Tennyson, R. D. (2001). Defining core competencies of an instructional technologist. Computers in Human Behavior, 17, 355-361.

Williams van Rooij, S. (2013). The career path to instructional design project management: An expert perspective from the US professional services sector. International Journal of Training and Development, 17(1), 33-53.

Wakefield, J., Warren, S., & Mills, L. (2012). Traits, skills, and competencies aligned with workplace demands: What today's instructional designers need to master. In P. Resta (Ed.), Proceedings of society for information technology and teacher education international conference 2012 (pp.3126-3132).

# data[].attributes.attachment_link.title
Standards and Competencies for Instructional Design and Technology Professionals

# data[].attributes.attachment_link.description
Students entering the field of instructional design must possess a wide array of competencies to be successful in their future roles (Ritzhaupt & Martin, 2014). Competencies are the knowledge, skills, and abilities professionals need in their roles, while standards speak to a pre-defined level of quality or attainment of those competencies.

# data[].attributes.headline
Developing and monitoring evaluation framework

# data[].attributes.subject
Developing and monitoring evaluation framework

# data[].attributes.attachment_link.provider_name
Social Care Institute for Excellence (SCIE)

# data[].attributes.attachment_link.readable_text
Why do we need an evaluation framework?

A well thought-out monitoring and evaluation framework will guide effective and systematic data collection and form the evidence base for assessment of progress and impact over time. It should be developed up front, as part of the programme design phase, to help clarify assumptions about how investments and initiatives are likely to deliver intended outputs, outcomes and impacts.

An evaluation framework captures key information about how progress and impact will be evaluated. There are different types of monitoring and evaluation, including:

point-in-time through to a programme of ongoing data collection and assessment

focus on a single project or a range of programmes

summative evaluation (to assess if outcomes and targets have been met)

formative evaluation (to identify how and why progress has been made)

formal audits, external evaluations, internal reviews or performance assessments.

Due to the developmental nature of integrated care, most evaluations will have a formative element - one that helps to understand what works and why - to build evidence for what should be scaled up, what should be improved and what should be decommissioned.

Evaluation planning

An evaluation framework - or matrix - sets out the plan for how to measure an outcome, as well as collect and analyse data. For each level, it sets out the aim, outcomes, measures, and a plan for data collection, analysis and reporting.

Specific aim

What are you trying to evaluate?

For example, was the programme effective?

Outcomes

What outcome is this regarding?

Measure

What is the measure that you are basing this evaluation on?

Is there an evaluation norm (i.e. standard to be met). For example, 80% of those accessing services.

Data collection

Is the data that is being collected relevant to stakeholders?

What source will be used?

Is the data already available? What approach will you take if it is not? For example, primary/ secondary research

Is baseline data available? What is your approach if baseline data isn't available?

What timeframe is the data collection based on (for example, weekly/monthly)? Is it consistent across all stakeholders?

What method are you going to choose to collect data (for example, survey)?

Who will be accountable?

Have you taken equal opportunities and ethical issues into account?

Do you have permission to use the collected data?

Is the collected data kept safe and confidential?

Analysis and reporting

How will the data be analysed (for example, annual evaluation)?

Are the selected methods manageable regarding resource requirements?

Who will scrutinise the data?

Which meetings/boards will this be reported to (including timeframe/ frequency)?

What changes will be made as a result?

Counterfactual analysis

How do we know what has made a difference? Counterfactual analysis enables judgements to be made about what changes are a direct result of the intervention(s). This helps establish which changes in outcomes are directly attributable to a project or a programme, versus those which would have occurred anyway. In other words, an analysis of the counterfactual takes an evaluation beyond just understanding whether outcomes have been achieved and allow an assessment of the extent to which observed changes in outcomes are a result of the intervention or of other factors.

While counterfactual analysis takes time and resources, it provides a stronger evidence base for decisions about whether there is value in continuing or scaling up an intervention. The Public Service Transformation Academy has an introductory guide on evaluation, a section of which is dedicated to understanding the counterfactual.

It highlights the following approaches.

Randomised controlled trials (RCTs)

These are the 'gold standard' of evaluation. Several people are randomly assigned to two or more groups to test a specific change. One group (the experimental group) receives the change being tested while the other (the control group) receives an alternative service or the existing service. RCTs are the most methodologically challenging form of evaluation, and need expert input to design and deliver.

Matched group comparisons

Where RCTs identify target and control groups and collect data over time, matched group comparisons use existing data sets to select similar groups or cohorts of patients or service users, some of which are users/recipients of the service evaluated and others who are not. Matched group analysis can also focus on comparing a geographical area with a similar area elsewhere. This often involves significant statistical analysis and may be difficult to design and deliver without significant evaluation expertise.

Experimental or quasi-experimental approaches

The basic structure of a quasi-experimental evaluation involves examining the impact of an intervention by taking measurements before and after it is implemented. This type of 'time series' analysis can be a very effective method and may not be as expensive to implement as other methodologies. It can be delivered without significant evaluation expertise.

What would have happened in the absence of your integration programme or project?

What would have happened had you done nothing?

Case study: Evaluating Sunderland's All Together Better Vanguard Programme (MCP) Open

The evaluation of Sunderland's All Together Better Multi-Specialty Community Provider Vanguard programme ran between November 2016 and June 2017. The evaluation, which was co-developed with specialist research and consulting organisation Cordis Bright, had a number of aims:

Consider the context of the Sunderland Care model

Review the Recovery at Home/Older People's Assessment and Liaison service

Review the Community Integrated Teams

Review the programme of Enhanced Primary Care

Consider the leadership and governance functions specific to Sunderland

Review the overall outcomes of the Vanguard programme, performance against expectations, and any unintended outcomes

A collaborative approach, based on the principles of action research, was taken to deliver the evaluation. All research approaches, methods and tools were agreed with an evaluation steering group before use in the field. The diagram below outlines the approach and evaluation activity undertaken.

A collaborative evaluation approach

Phase 1. Baseline report and evaluation framework (Dec 2016)

Project launch

Review of documentation

Literature review of 'what works' in implementing an MCP

In-depth interviews with key senior stakeholders

Deliver draft baseline report and evaluation framework

Circulate and meeting with steering group for feedback

Evaluation framework and baseline report signed off

Phase 2. Implementation of evaluation framework (Jan-Mar 2017)

Review of documentation, performance management and budget information

In-depth interviews with stakeholders

e-Survey of staff and stakeholders

Analysis and reporting

Delivery of draft final report

Phase 3. Delivery of final report and dissemination (Apr-Jun 2017)

Circulate draft report for feedback

Sense-testing workshop with senior stakeholders

Finalise and sign off final report

Presentation of report findings

Key learning

Collaborative approach meant that the evaluation addressed and answered the key evaluation questions

Development of a clear SMART evaluation framework meant that all evaluation stakeholders understood the key evaluation questions, activity and outputs as well as how process, impact and outcomes were going to be demonstrated and evidenced

Sense-testing workshops with stakeholders helped to ensure a set of grounded, practically useful recommendations which also increased 'ownership' and 'buy-in' among stakeholders

The final report can be accessed here: Sunderland All Together Better (MCP)

When is evidence 'good enough'?

The quality of evidence varies depending on the evaluation design and methodology. At times, it is not possible to do more extensive impact evaluations, yet evidence about progress and impact is needed to report on progress and drive decision-making. While not as robust as more systematic evaluations, other less structured forms of evidence can be useful. These include case studies, surveys, stakeholder interviews, peer or stakeholder review sessions, point-in-time data analysis and literature reviews (of learning from implementation of similar initiatives elsewhere).

The example of a Better Care Fund programme review in Hounslow illustrates how good quality information is gathered without undertaking a full impact evaluation.

Case study: Point-in-time programme reviews - Hounslow Open

At times it is not possible to undertake a full evaluation over a longer period. Point-in-time reviews enables leaders to take stock on a project or a programme, and identify learning about what works well and what works less well. In 2016, leaders in Hounslow decided to undertake a review of all their BCF schemes. The straightforward but systematic approach can be used in a number of different settings.

The aim of the review was to inform a revised approach to the BCF in 2016/17. Over a period of five weeks, the key stakeholders involved in the BCF came together to explore what was working well and where there were issues and performance challenges.

The review aimed to answer the following key questions:

Are the BCF programmes delivering what they set out to deliver?

How does the BCF need to evolve to reflect what is happening on the ground?

How can we maximise the impact of the schemes that are working for 2016/17?

How can we ensure that the BCF effectively enables the vision for whole-systems integration?

How can the BCF be best used to support the Hounslow health and social care economy?

In addition to the workshops with key stakeholders, project leads, staff and other stakeholders were interviewed and performance data and existing reviews were collated. An analytical framework was used to map all the evidence gathered for each project against the overall goals of the BCF plan.

Against the analytical framework below, schemes were rated on a scale of 1 - 10 where 1 is 'not at all' and 10 is 'to a great extent'.

Measure (1-10)

Scheme one

Scheme two

Scheme etc

The results of the qualitative assessment and the costs of each scheme were mapped against a cost/impact matrix as set out below.

The findings were used to inform decisions about what schemes should continue and be scaled up (schemes with highest impact and lowest cost scores) and which ones should be improved or decommissioned (schemes with lowest impact and highest costs).

In summary: top tips to develop a monitoring and evaluation framework

Consider evaluation up front. What information is needed about progress and impact? How will the information be used?

Be systematic. Providers should draw up an evaluation plan or framework to set out the aims, expected outcomes, measures and targets and a plan for how they will to monitor and analyse the results.

Be realistic. It is not always possible or practically feasible to implement a bestin- class evaluation framework. Instead, partners should think carefully about which key measures could be used to monitor progress.

Flexibility is important. Include both qualitative and quantitative measures and consider what the evidence is indicating as the evaluation progresses. It is important to be able to adapt and change an approach over time, if required.

Take time to understand the data.

Download

All SCIE resources are free to download, however to access the following download you will need a free MySCIE account:

Available downloads:

How to understand and measure impact of integrated care

# data[].attributes.attachment_link.title
Developing a monitoring and evaluation framework

# data[].attributes.attachment_link.description
An evaluation framework captures key information about how progress and impact will be evaluated.

# data[].attributes.attachment_link.description
Program evaluation looks at the parameters, needs, components, and outcomes of program design with an eye towards improving student learning. It involves a complex approach, taking into consideration needs assessment, curriculum mapping, and various models of program review.

# data[].attributes.attachment_link.readable_text
Where Evaluation Fits into the Instructional Design Process

Ross and Morrison (2010) sum up three high-level types of evaluations quite nicely:

‍ Formative evaluation is used to provide feedback to designers as the instruction is "forming" or being developed. Summative evaluation is conducted to determine the degree to which a completed instructional product produces the intended outcomes. Confirmative evaluation examines the success of instruction as it is used over time.

‍ If you have the resources to conduct all three assessment types, there's no question about where evaluation fits into the design sequence. It happens throughout the entire process (and continues after design is done, as well).

Of course, not every company will be able to run the large number of assessments necessary for formative, summative, and confirmative evaluation of every course. In these cases, determining when to evaluate your instructional design may come down to your goals.

To make sure that your design is effective when it's debuted, for example, formative evaluation practices are useful. Summative evaluations determine how much employees learned and whether this knowledge creates behavior change. And confirmative evaluations measure the long-term success of the program.

Frequent evaluation allows for continuous improvement. So it's best to assess your instructional design as often as you can.

Designers who follow the ADDIE model might be tempted to leave all evaluation for the end of the process. But evaluating more often has many benefits.

Play to your organization's strengths here. If you have a budget big enough for multiple evaluations throughout the process, go for it. If not, you may need to get creative. We'll talk more about these three types of assessment a bit later.

Instructional Design Evaluation Strategies: The Kirkpatrick Model

How, then, do we measure the effectiveness of instructional design? There are many theories, each with advantages and disadvantages.

One of the most useful frameworks for evaluating your instructional design is the Kirkpatrick model. Although it has drawbacks, the model's simplicity and popularity make it a good option for companies looking to evaluate their instructional design. Especially if you're interested in comparing results to other companies in your industry. The model has four levels:

Let's take a look at each one individually.

1. Reaction

How do learners react to the training? If their reaction is positive, they're more likely to have positive learning outcomes. A negative reaction doesn't preclude learning, but it makes learning less likely.

The factors that elicit positive reactions vary. Trainers, training methodologies, and audiences all have unique needs and preferences. But there are some commonalities. A 2009 study found that trainees' perceptions of the training's efficiency and usefulness, as well as the trainer's performance, were correlated to reactions.

The usefulness of the training was the most important factor in the study. Which won't surprise anyone who's ever been to an irrelevant or useless training.

Trainings that are efficient (i.e., don't waste time), useful, and conducted by likable, interesting, effective trainers are going to get the most positive reactions. And that will help boost learning.

(Though it's important to note that there are multiple types of reactions to instruction, and not all of them may be correlated with learning.)

2. Learning

How much did participants learn? Did they retain the information over time? These questions don't seem overly complicated. But getting reliable objective measures can be difficult.

For example, when measuring learning, you need to know how your participants' level of knowledge before they completed the training. If trainees were already experts, showing that they're experts at the end of the training isn't very interesting. Which is why both pre- and post-testing is necessary for effective measurement of learning.

There are many ways of testing the acquisition of knowledge; assessments, self-assessments, informal testing, and so on. Choosing the right instrument may depend on the type of training you're running and your measurement objective.

Knowledge acquisition, for example, can be measured with a simple multiple-choice quiz. The ability to apply that knowledge, however, is more difficult to measure, and may require a more in-depth assessment.

‍

3. Behavior

Are your participants applying what they learned to their jobs? This is possibly the ultimate test of the effectiveness of instruction.

Successfully measuring behavior change isn't easy. You'll need to determine the most relevant behaviors, measure them before the instruction, measure them after the instruction, and figure out if the training caused the change.

Let's look at an example. If you run a course on sales enablement, the employee behavior you're trying to affect might be the sharing of information between marketing and sales departments. You'll need to identify some sort of measurable metric to see if this is actually happening. Which metrics might you look at to assess behavior change in sales enablement? Many knowledge-sharing systems provide analytics on content usage, like the number of times a particular piece of content has been accessed or used in a sales pitch. How many sales interactions include materials developed by marketing? That's a metric that you can compare pre- and post-test.

Other software platforms can provide you with similar metrics. And while behavior change can be measured independently of tools, those instructional design tools make the process much easier.

While they aren't as rigorous as objective measures, employee surveys are useful as well. In our case, we might ask employees how often they use the sales enablement system, the percentage of documents they think are shared between the two groups, and similar questions.

The most important thing to remember here is to choose a metric that's closely tied to the behavior you're trying to influence. Choosing the wrong measure of behavior can skew your results.

4. Results

Did your training affect the business's bottom line? In the end, this is the most important question. If your training increases revenue, decreases costs, or otherwise improves profitability, it was a success.

To answer this question, you need to extend measures of behavior change to business results. If your sales and marketing teams are communicating more effectively, and those teams are pulling in more money, it's a good bet that your training had a positive bottom-line effect.

There's one snag, though: how do you know that the bottom-line change had anything to do with the training? There are countless factors at play when it comes to revenue and profitability, and isolating a single variable is tough.

Your sales team might be selling a lot more after the sales enablement course, but how do you know that a manager didn't switch up their management style as well? Or that a new employee became a selling superstar? These questions are difficult to answer.

The best answer, unfortunately, is that you'll need to use your reasoning skills. Look at as much information as possible and see if there might be confounding variables. If you find one (or more), see if you can control for them. If you can't, you may just have to draw conclusions based on the available data.

Measure What's Important

Kirkpatrick's model is a useful framework for evaluation. But the specific model you use isn't important. What's important is that you measure the effectiveness of your training and take actions based on what you find.

And when it comes to measurement, the most important factor is bottom-line results. Is your instruction having a positive impact on the bottom line?

Unfortunately, many companies don't know. According to Patel (2010),

90% of surveyed companies measured participant reactions

80% measured student learning

50% measured behavior

40% measured results

Instructional designers are measuring what's easy to measure, but not what's important. Remember what you're ultimately trying to achieve when you set out to evaluate your instruction.

Formative, Summative, and Confirmative Evaluation Methods

In addition to knowing what you should be measuring, it's important to understand why you're measuring it. We previously discussed formative, summative, and confirmative evaluation. Now we'll talk about what those are and why they're important.

Formative Instructional Design Evaluation Methods

Formative evaluation can happen at any point during the learning process. It can take place while the instructional materials are still being developed, while learners are using them, or after the training is done.

The point of formative evaluation is to help instructional designers improve their instruction materials and methods.

Rapid prototyping and testing of instructional materials often happens alongside formative assessment before the training has been implemented. Because formative evaluation is best done quickly, the methods for this type of assessment are often somewhat informal.

These methods might be as simple as asking people what they thought of the training. Was the material clear? Was it presented efficiently? Did it seem useful? Were the practical exercises helpful? Questions like these help designers optimize their trainings to maximize engagement and learning.

Most of the questions in formative evaluations are related to Kirkpatrick's reaction and learning steps. If you're getting positive reactions and trainees are learning, you're on the right track.

One-on-one interviews, focus groups, surveys, and informal conversations are useful for formative instructional evaluation. Field trials are also useful in that they give designers an idea of how the training will be conducted in real life, which may get different responses than more controlled trials.

Summative Instructional Design Evaluation Methods

Summative evaluation happens after the training has been completed. The primary question that summative evaluations ask is whether the training had the intended outcome. Did participants learn the information the training was designed to teach?

If they did, the training was a (summative) success.

Determining whether participants learned the information is relatively easy. Post-test quizzes and questionnaires give designers insight into the efficacy of the training (of course, this requires a solid pre-test regimen as well). These instruments are great for measuring the knowledge uptake from the training.

This corresponds to the learning portion of Kirkpatrick's model. But summative assessment can also take behavior change into account.

Figuring out if learners are modifying their behavior is more difficult. As we discussed above, this may be best accomplished with tools that measure the metrics you're interested in. Knowledge-sharing software, customer service apps, project management platforms, and other types of tools have analytics and metrics built in and can help managers track changes in performance.

Surveying managers might also be an effective method, especially if the managers weren't a part of designing or implementing the training. Choosing the right summative metrics is up to the designer. Be sure to give them a lot of thought; you might come up with an obvious metric at first but discover a more apt one later on.

Confirmative Instructional Design Evaluation Methods

As we've discussed, measuring the bottom-line effects of a training is the ultimate determination of whether the instruction was valuable. Confirmative evaluation takes a long view and asks whether the training was effective weeks, months, or even years after it was completed.

Confirmative evaluation benefits from many of the same methods as summative evaluation, but designers will also be expected to report on the ultimate business metric: return on investment (ROI). If a training has a positive ROI, it was a success. This evaluation is all about the results step in Kirkpatrick's model.

Of course, that means that there needs to be a monetary value associated with the outcomes from the training. And that will likely require the cooperation of many people in different areas of the company. If your training improved manufacturing processes, you'll need information from the manufacturing team to see if the practices and concepts learned in the course are still being applied.

You'll also need accounting information to find out if the manufacturing team is more profitable than it was before the training. You may need to use statistical analysis to find out if there's a significant correlation between training-related behavior change and positive financial outcomes for the company.

And it's possible that there will be intermediate steps, as well. If the instruction was aimed at reducing injuries, you'll need to find out how much the average injury costs your company-then look at behavior change to determine how many injuries were averted by the training.

While both formative and summative assessment can benefit from qualitative standards, confirmative evaluation is all about numbers. Which is why you'll need hard financial data to compare to information on behavior change.

Begin the Evaluation Process Early

If you're just starting your learning and development program, this might all sound overwhelming. But remember that evaluation is an ongoing process that runs alongside instructional design.

You don't need a fully fledged evaluation system for your entire program at the outset. Instead, develop it as you go along. Start with the goals of your training, and develop formative, summative, and confirmative measures that tell you whether you're heading toward those goals.

Like anything else, effective evaluation takes practice and experience. It can be tempting to give scant attention to instructional evaluation, because doing it well takes a lot of time. But with the right system in place, your evaluation efforts will get faster and you'll post a better ROI.

And isn't that worth taking the time to do?

# data[].attributes.attachment_link.title
How to Conduct an Instructional Design Evaluation - Continu

# data[].attributes.attachment_link.description
Effective instructional design evaluation proves the value of training. Use this checklist to see how yours holds up.

# data[].attributes.attachment_link.readable_text
This is Iman Khamis Albusaidi. I am an English teacher who has been teaching since 2009. I am proud to be a master student in ILT department in Sultan Qaboos University.
I am passionate about integrating modern technology in my lessons as a significant tool to enhance learning and teaching experience. I am excited to share my academic journey with you.

Technology has always been at the forefront of human education. From the days of carving figures on rock walls to today, when most students are equipped with several portable technological devices at any given time, technology continues to push educational capabilities to new levels. In looking at where educational methods and tools have come from to where they are going in the future, technology's importance in the classroom is evident now more than ever

# data[].attributes.attachment_link.description
This is Iman Khamis Albusaidi. I am an English teacher who has been teaching since 2009. I am proud to be a master student in ILT department in Sultan Qaboos University. I am passionate about integrating modern technology in my lessons as a significant tool to enhance learning and teaching experience.

# data[].attributes.attachment_link.readable_text
Starting this site with the beginning of my masters' degree in instructional and learning technology at Sultan Qaboos University. It will be a collection of my work as learning reaches a higher level with sharing. Enjoy your time scrolling and don't forget to teach me as well!!

# data[].attributes.attachment_link.readable_text
I am Afrah Rashid Aziz Al Habsi. Welcome in my e-portfolio where I share with you my experience as a master student in Learning and Studying Technologies in SQU. I am proud to share examples of my greatest work. I hope you'll enjoy viewing my projects and all of the unique content I create as much as I enjoyed working on them. Take a moment to check out the site and get in touch with any questions.

To play, press and hold the enter key. To stop, release the enter key.

# data[].attributes.attachment_link.description
I am Afrah Rashid Aziz Al Habsi. Welcome in my e-portfolio where I share with you my experience as a master student in Learning and Studying Technologies in SQU. I am proud to share examples of my greatest work.

# data[].attributes.attachment_link.readable_text
Welcome to My

e-Portfolio

Hello and welcome to my e-portfolio. My name is Thuraiya Al Saadi. I am a Master's student studying in the Instructional and Learning Technology department at Sultan Qaboos University.

# data[].attributes.attachment_link.description
Welcome to My e-Portfolio Hello and welcome to my e-portfolio. My name is Thuraiya Al Saadi. I am a Master's student studying in the Instructional and Learning Technology department at Sultan Qaboos University.

# data[].attributes.attachment_link.readable_text
Introduction:
I drew in my mind a dream more than ten years ago to become a lecturer at Sultan Qaboos University, and now I have taken the first step in this dream, now I started studying a Master's in Teaching and Learning Technology at Sultan Qaboos University, and I hope to reach my dream as soon as possible.

# data[].attributes.attachment_link.description
Introduction: I drew in my mind a dream more than ten years ago to become a lecturer at Sultan Qaboos University, and now I have taken the first step in this dream, now I started studying a Master's in Teaching and Learning Technology at Sultan Qaboos University, and I hope to reach my dream as soon as possible.

# data[].attributes.attachment_link.readable_text
I am Afrah Rashid Aziz Al Habsi. Welcome in my e-portfolio where I share with you my experience as a master student in Learning and Studying Technologies in SQU. I am proud to share examples of my greatest work. I hope you'll enjoy viewing my projects and all of the unique content I create as much as I enjoyed working on them. Take a moment to check out the site and get in touch with any questions.

To play, press and hold the enter key. To stop, release the enter key.

# data[].attributes.attachment_link.description
I am Afrah Rashid Aziz Al Habsi. Welcome in my e-portfolio where I share with you my experience as a master student in Learning and Studying Technologies in SQU. I am proud to share examples of my greatest work.

# data[].attributes.headline
Developing evaluative judgment and self-regulated learning perspective

# data[].attributes.subject
Developing evaluative judgment and self-regulated learning perspective

# data[].attributes.headline
Fusing self-regulated learning and formative assessment

# data[].attributes.subject
Fusing self-regulated learning and formative assessment

# data[].attributes.headline
Checklist and rubrics for evaluating online programs

# data[].attributes.subject
Checklist and rubrics for evaluating online programs

# data[].attributes.attachment_link.readable_text
by Sue Bauer,

"Ensuring quality and success for your online and blended courses, online programs, and initiatives through the use of quality checklists and rubrics exhibits that your institution has a compliance process, addressing critical legislation and accreditation standards." -Amanda Major, Community of Practice Discussion

Overview

This section covers checklists, rubrics, and guidelines for evaluating online programs and course quality. The Selection/Application/Implementation Guide and Resources/Examples may be used to create a customized program and course evaluation process for an institution's specific needs. These artifacts may be used in a variety of contexts such as self-assessment, instructional designer review, peer review, institutional online program effectiveness, and accreditation standards.

1. Identify fundamental, critical, and essential elements for your institution's online courses, programs, and institutional initiatives.

2. Use samples and resources to identify factors and elements related to the importance of your evaluation for high quality online course development and delivery.

3. List, categorize, and prioritize developed guidelines into personalized checklists, rubrics, or guidelines for your institutional needs, requirements, and implementation.

Note: If you are interested in more information on program evaluation and effectiveness, visit the Evaluation page where you will find more detailed information, tips and resources.
Selection/Application/Implementation Guide

This section presents recommendations that may be utilized when analyzing, selecting, and applying criteria to the institution's assessment method for effective online programs, course requirements, and sometimes to meet accreditation standards. How will "quality" and "success" in online learning be defined at your organization? The first recommendation is to start at a higher level of prioritizing needs. What national or institutional needs must be met for your institution? This could include matters such as copyright, Family Educational Rights and Privacy Act (FERPA), and online accessibility protocols. Along with institutional criteria, departments and programs may also have specific criteria for online course content. Program criteria is a much more specialized area to develop and although some patterns may emerge, particular criteria may be highlighted and applied to specific courses and programs.

The overall process of analyzing and selecting criteria for online course/course program development can be narrowed down to the determination of specific course and content requirements. All courses will generally include a syllabus and content for delivery.

Elements deemed critical for these areas should also be listed and implanted in the evaluation piece. Once the above levels of criterion have been considered and listed, it is recommended that the criteria be categorized and prioritized for your application.

Considerations for Implementing Rubrics, Checklists, and Guidelines

Although all of the listed criterion are important and should be highlighted, prioritizing can assist faculty in their process for online content development. Once there is a comprehensive list of institution criteria, it is time to develop the rubrics, checklists, or guidelines. Consider how and when the criteria will be used. It is very probable that more than one of these rubrics, checklists, or guidelines will be created to meet your development needs. Ask yourself the following for context and usage:

Will these rubrics, checklists, and guidelines be used to assess courses during the delivery of your professional development course?

Will they be used for self-assessment, an instructional designer review, a peer review activity, or all of the above?

Will you develop the rubrics, checklists, and guidelines as formal assessments that will prohibit or enhance an instructor's ability to teach online?

Will they be used more informally as self-assessment or peer review activities?

Will elements be included to assess the course's effectiveness after the delivery of the course online?

Although this section may seem an overwhelming list of considerations, all are important in the development of your criterion for online course effectiveness. It is critical/imperative to decide what is important at your institution and how it will be clarified to instructors in your professional development course.

Resources/Examples

This section provides high quality examples of rubrics, checklists, and guidelines used to assess online programs and course evaluation/delivery standards. These resources were developed and utilized by other institutions. It is suggested that you reference these items, as appropriate, while you develop your own list of standards. They may also be considered "as-is" for adoption at your institution.

Program Evaluation Methods National Standards for Quality Online Learning
This document, the International Association for K-12 Online Learning's (iNACOL) National Standards for Quality Online Programs, is the third of iNACOL's online education standards, following the National Standards of Quality for Online Courses and National Standards for Quality Online Teaching.

Course Evaluation Methods
Florida Online Course Design Quality Designations
With the aim of increasing the number of quality online courses in Florida, the Board of Governors Quality Workgroup provided guidance for an online course design quality review processes flexible enough to span contexts of all State University System and Florida College System higher institutions. Online courses vetted through an online course design quality review will be designated as "Quality" in the FloridaShines course catalog. Reviews can result in a "Quality" or "High-Quality" designation based on the Quality Matters (QM) Higher Ed with a process unique to Florida.

The Florida initiative uses QM as a default but also, through an "Institutional Equivalency Process," allows individual institutions to have their own institutional standards and processes approved for use instead. The Florida State University System, through its 2025 Strategic Plan for Online Education, has called for a state-wide initiative for recognizing "quality" and "high quality" online courses.

OLC Quality Course Teaching and Instructional Practice (QCTIP)
Designed to evaluate the overall classroom experience, this tool can be used to evaluate teaching effectiveness in several areas, including: Course Fundamentals, Learning Foundations, Faculty Engagement and Student Engagement.

Note: Access to this rubric may require OLC Membership.

UCF Quality Course Review
UCF High Quality Course Review

In support of Florida's state-wide quality initiative, UCF's Center for Distributed Learning developed a tiered system for reviewing courses with 2 designations: Quality and High Quality. The standards align with Quality Matters (QM) and additional standards have been added to address UCF's specific needs.

Sample Course

The TOPkit sample courses have additional information regarding the rubrics used to assess courses and modules during a peer-review activity included in the courses.

Recommended Articles

Beyond the above list of resources and examples, listed below are addition readings that may assist in the development of your campus resources.

Brown, V., Lewis, D., & Toussaint, M. (2018).

Thompson, K. (2016).

Lee, C., Dickerson, J., Winslow, J. (Spring 2012).

McGahan, S., Jackson, C., Premer, K. (December 2015).

Office of Educational Technology. (November 2014).

Shelton, K. (2010).

The Institute of Higher Education Policy. (April 2000).

References

Bauer, S. & Sugar, A. (2012 April). Professional Development for Preparing Online Faculty. Conference Presenter at USDLA 2012 NATIONAL CONFERENCE, Hilton St. Louis, St. Louis, MO. Retrieved from https://sites.google.com/site/idl6543redesign/

Michigan Virtual University (2012). Planning Guide for Online and Blended Learning. Retrieved from https://micourses.org/resources/pdf/toolkit/MVU_RPT_PlanningGuide.pdf

Sue is an instructional designer with the Center for Distributed Learning at the University of Central Florida. Sue collaborates with faculty to deliver professional development opportunities, apply best practices in course development, and offer consultations and instructional design guidance. Sue's special interest areas are project management for instructional designers in higher education, mobile teaching/learning, and MOOCs.

# data[].attributes.attachment_link.title
Checklists & Rubrics for Evaluating Online Programs

# data[].attributes.attachment_link.description
by Sue Bauer, "Ensuring quality and success for your online and blended courses, online programs, and initiatives through the use of quality checklists and rubrics exhibits that your institution has a compliance process, addressing critical legislation and accreditation standards." -Amanda Major, Community of Practice Discussion This section covers checklists, rubrics, and guidelines for evaluating online programs and course quality.

# data[].attributes.headline
Existing criteria determining course quality in DE

# data[].attributes.subject
Existing criteria determining course quality in DE

# data[].attributes.attachment_link.readable_text
Today, while governments mostly focus on indicators such as productivity, cost effectiveness, social satisfaction, and accountability as quality indicators, higher education institutions take into account fundamental indicators such as student output, learning and teaching processes, student and staff support, assessment, evaluation, and resources. These fundamental indicators can be varied institutional-based, program-based or course-based. In the context of higher education, quality assurance of distance education actually requires different quality indicators and different regulations from traditional higher education. On the other hand, quality assurance in distance education is more debated than traditional education. Currently, people do not trust the quality of distance education. Therefore, initiatives to increase the quality of distance education may be increased the assurance in distance education. The aim of this study is to examine the studies which provide criteria to evaluate online courses for creating or improving quality standards in online environments in the context of higher education and also to make a descriptive analysis by presenting differences and similarities between the course quality measures of these studies. This study is expected to guide the online course developers, faculty, and administration of higher education institutions to improve or to redefine the quality of their proposed online courses. INTRODUCTION Quality assurance is designed to improve and prove the quality of methods, educational products and outputs in an organization. A quality assurance system includes standards to be awarded and documented procedures for all defined processes, as well as defined ways to respond to a number of problems, and significant accountability for outputs (Kirkpatrick, 2005). Quality assurance in online distance learning requires different quality indicators from traditional education, as well as evaluations in different settings, and also readiness for e-learning (Stella, Gnanam, 2004; Latchem, 2014). Different views on quality assurance in open and distance learning are pioneers to the development of pedagogical paradigms in open and distance learning. These different views are summarized as follows (Latchem and Jung, 2012: 13-14): (1) distance education and face-to-face education should be assessed with the same criteria and methods, (2) the quality assurance criteria and mechanisms used in traditional education cannot be applied to open and distance learning, since aims, institutional structure, registration, procedures are different, (3) certain guidelines and standards are required for e-learning, (4) the basic principles of learning and teaching do not change at open and distance learning apart from technology, (5) quality assurance in open and distance learning must be compulsory, accountable and managed externally, (6) quality assurance in open and distance learning must be optional, runs internally and should be developed a culture of institutional quality, (7) it is the duty of the providers of open and distance learning to prove that the processes and outputs of open and distance learning are at least as good as traditional institutions, (8) traditional learning and teaching methods are old fashioned, more learning-centered, so structural and connectivist methods should be applied and then the most appropriate one must be selected from practical, face-to-face or technologically enhanced learning methods. In the light of these discussions, we can say that there is a consensus on the idea that " online courses require additional attention to detail " .

# data[].attributes.attachment_link.title
EXISTING CRITERIA DETERMINING COURSE QUALITY IN DISTANCE EDUCATION

# data[].attributes.attachment_link.description
Today, while governments mostly focus on indicators such as productivity, cost effectiveness, social satisfaction, and accountability as quality indicators, higher education institutions take into account fundamental indicators such as student output, learning and teaching processes, student and staff support, assessment, evaluation, and resources. These fundamental indicators can be varied institutional-based, program-based or course-based.

# data[].attributes.attachment_link.readable_text
The Academic Life in Emergency Medicine (ALiEM) Faculty Incubator was hard at work during the pandemic to bring you the fifth volume of the Education Theory Made Practical series. This series strives to make theory accessible to educators by distilling the background and key literature of each theory and grounding them in practical education scenarios.

The Faculty Incubator is a year-long professional development course for medical educators centered around a virtual community of practice (a concept we have all started to appreciate during quarantine). Teams of 2-3 participants from around the world authored primers on education theories and different teams offered a first round of peer review on each post. As in prior years, they will be serialized on the ICE Blog for review and comment.

They have published three e-book compendiums of this blog series (, , ) and you can find the Volume 4 posts here (the e-book is in progress!) As with the previous iterations, final versions of each primer will be complied into a free eBook to be shared with the health professions education community.

Your Mission if you Choose to Accept it:

We would like to invite the ICE Blog community to peer review each post. Your comments will be used to refine each primer prior to publication in the final ebook. No suggestion is too big or small - we want to know what has been missed, misrepresented, or misconstrued. Comments as small as grammatical errors all the way to new scenarios for practical applications or new citations are welcome. (Note: The blog posts themselves will remain unchanged.)

Main Authors or Originators: Edward A. Suchman

Other important authors or works: Joseph S. Wholey, McLaughlin J.A. and Jordan G.B.

Part 1: The Hook
Sarah has just finished her ultrasound fellowship and is working in the Emergency Department at her new hospital. She has discovered that many of her new partners are not familiar with or comfortable using bedside ultrasound in clinical practice. When she inquires about this, many of her coworkers mention that they were educated prior to 2006, when ultrasound became incorporated into residency training as part of the required curriculum for residents. Sarah would like to design a training program for her colleagues to help them become more comfortable with performing and interpreting ultrasound in clinical practice. Initially, she would like to develop a curriculum to teach her colleagues how to perform and interpret basic bedside ultrasound studies and then expand to other imaging applications. What activities can be planned and what outcomes could be measured to ensure success of her program?
Part 2: The Meat

Overview

The logic model is a conceptual tool that can be used for program planning, implementation, and evaluation. This tool is designed to examine a program's resources, planned activities, and proposed changes or goals in an organized fashion. It describes the linkages between resources, activities, outputs, outcomes, and their impact on the program as a whole. It provides a model of how a program's component parts might function together. []

The logic model is represented visually in four main sequential components: inputs, activities, outputs, and outcomes. These comprise two main domains: planned work and intended outcomes. Planned work includes inputs and activities while intended outcomes reflect the outputs and outcomes. Outcomes can be measured as immediate, intermediate, and long-term. Some sources also suggest a fifth component, a measurement of impact, at the end of the model in lieu of, or in addition to, long-term outcomes. [] In business applications, many logic models also show external influences as arrows into each of the components to show how each of these external factors affects each of the steps of the model.

This model has been used in designing and planning new programs, or when it's time to evaluate a program and restructure. It provides a structured framework to systematically evaluate program components and to facilitate communication with team members, and helps guide the acquisition of the information necessary for decision making. It is specifically useful in determining evaluation and management strategies for medical education programs. []

Logic model components, including inputs, activities, outputs, and outcomes. From AMEE 67 []

Background

The early foundations for the logic model were first set out in a 1967 book by Edward A Schuman about evaluative research. [] The concept of a logic model was previously captured in other structures and under different variations including "Chains of Reasoning," "Theory of Action," and "Performance Framework."[ 1] Bickman (1987) introduced logic models as a tool for program evaluation that emphasized program theory, the idea that interventions contribute to a downstream chain of results and outcomes. []

The first publication using the term "Logic Model" was by Joseph S Wholey (1983). [] The first developers of the logic model came from business, public sectors, and international non-profit sectors. Logic models did not become widely used until United Way published Measuring Program Outcomes in 1996. [] This article was important in establishing the terms and structure used today for developing Logic Models.

The W.K Kellogg Foundation published a widely available Logic Model Development Guide which has been used for public policy and healthcare planning.[] Over the last decade, logic models have also been used for medical education program evaluation.

The logic model contains four key components. First, inputs can be seen as the resources necessary to operate the program. [ 3] Inputs include resources dedicated to or consumed by the program and can include financial resources and funding, protected time for faculty or staff, expertise of faculty and staff, administrative support, and physical resources such as facilities and equipment. For example, a department running an EKG curriculum for residents could list talented EM and cardiology faculty, experienced ED techs, and reserved time in the departmental conference room among its inputs.

Inputs are then used to operate planned activities.[ 3] Activities represent what the program does with the inputs to fulfill its mission. Activities may include any combination of needs assessments, teaching, curriculum design, planning of sessions, faculty development, development of systems, or performance evaluations. These activities are dependent on the program's mission. For the EKG curriculum, the activities might include didactic sessions reading EKGs and time in the department placing EKG leads and interpreting them in real-time.

If the planned activities are accomplished, then the program will create outputs. [ 3] The outputs describe the direct, measurable products of the activities. This can include demographics such as number of participants in a program's activities, number who completed a certain curriculum, or program metrics such as number of programs, time in existence, or number of graduates of a program. In our EKG example, the outputs might be reaching 24 residents, or conducting 12 sessions in a year.

Outcomes look at the benefits for participants during and after program activities. Outcomes can be measured immediately, as well as in the intermediate and long-term time frames. These can include increased knowledge or skill, satisfaction with quality of activities, or improvement in course evaluations. Depending on the program, clinical, teaching, or academic success may be appropriate outcomes, including awards, or productivity. For the EKG curriculum, outcomes might include residents attaining a higher score on a test of EKG proficiency, or self-reported improved confidence with EKG interpretation.

Finally, if the benefits of the program are achieved, then the activities implemented as part of the program will have an impact on external factors such as an organization or system. [ 3] This can relate to impact on the community, environment, or infrastructure. For the EKG example, this might translate to improved rates of appropriate recognition of ST-elevation myocardial infarctions, or improved functional outcomes for patients.

Modern takes or advances

The logic model is a tool that can facilitate communication and idea sharing, identifying the necessary components that are critical to the successful execution of a program. The logic model can help identify if there are implausible linkages among program elements or redundant pieces.[ 1] Benefits of the logic model include gaining a common understanding and expectations of resources, their allocation, and expected results. In the past few decades, the logic model has been applied to various applications in medical education and healthcare.

Program managers use the logic model to argue how or why the program is meeting a specific customer need, whether that customer is in a private sector or a medical learner. [ 1] The logic model is also used to facilitate thinking through faculty development and other large-scale education initiatives. [ 3] It has also been adopted on a larger scale for healthcare systems innovations, [ 1] or in the public health workforce to support communication between divisions and ongoing program planning and evaluation. [] Portfolio evaluation, or the evaluation of multiple projects with a common purpose, also benefits from the use of logic modeling as a visual tool. []

Logic models have been used to create consensus among leaders or with stakeholders, as all parties can examine both the required inputs and the desired outcomes and how they will be measured. This approach can also be applied to medical education in the setting of institutional self-review. [] The World Federation for Medical Education utilized a logic model applied to further define and evaluate each of their accreditation standards. In this specific case, the logic model was used both for standard setting and consensus of their standards, but also as an evaluative tool. []

Other examples of where this theory might apply in both the classroom & clinical setting

Van Melle et al proposed the use of the logic model for program evaluation for competency-based medical education (CBME). [,,] The authors provide an outline of how to use a logic model to focus CBME program evaluation in a residency program, how to make program evaluation scholarly, and how to build capacity for program evaluation. In this case, the desired proximal outcome is enhanced resident readiness for independent practice and the desired distal outcome is improved patient care. [16]

The logic model has also been utilized by medical education programs as a tool for planning initiatives. Campbell et al described the model as a tool to guide the creation of a Fellows' College, or a centralized educational program for all pediatric subspecialty fellows at a single institution. [] They describe the inputs such as leadership, funding, individual subspecialty program structure, and existing curricula. They discuss activities such as an expanded curriculum, a mentorship program, scholarship program, and networking. The outputs were the number of fellows and clinical educators engaged in the Fellows' College. Finally, their outcomes measure success of the intervention and subsequent program modifications.

Annotated Bibliography of Key Papers
McLaughlin, J.A. and G.B. Jordan. Logic models: a tool for telling your programs performance story. Evaluation and Program Planning, 1999. 22(1): p. 65-72. [1]
This paper was one of the first to outline in detail the practical applications of the logic model. It was intended to explain to program managers in the public and private sectors how to measure and evaluate a business program and how to use that knowledge to improve a program's effectiveness. By utilizing clearly outlined figures and tables, these authors provided a detailed explanation on how to build a logic model for business managers.
Model Development Guide, ed. W.K.K. Foundation. 2004. [9]
The Kellogg foundation document outlined important definitions of the logic model as a method of program evaluation, defining it as, "a picture of how your organization does its work- the theory and assumptions underlying the program. A program logic model links outcome (both short and long term) with program activities/processes and the theoretical assumptions/principles of the program."
Otto et al Otto, A.K., K. Novielli, and P.S. Morahan. Implementing the logic model for measuring the value of faculty affairs activities. Acad Med, 2006. 81(3): p. 280-5. [3]
Otto et. al (2006) published the first article in Academic Medicine suggesting the use of logic models in medical education. They suggested the use of logic models for measuring how faculty development offices contribute to the recruitment, retention, and development of teaching faculty. They give an example of its use in a visual format with a comprehensive associated list of components in each category. Use of the logic model is suggested to facilitate the process of thinking through the entire faculty development process.
AMEE G Frye, A.W. and P.A. Hemmer, Program evaluation models and related theories: AMEE guide no. 67. Med Teach, 2012. 34(5): p. e288-99.
This paper takes a broader view of various program evaluation models. It is useful because it compares several different models. In addition to the logic model, it also discusses the experimental/quasi-experimental model, the Context/Input/Process/Product Model (CIPP model), and Kirkpatrick Model. It delves into the strengths and weaknesses of each type of model and how they can be applied to medical education. The authors in this paper provide a good description and analysis of each of the four essential elements of the logic model. It provides various medical education-centered examples of each of these elements.

Limitations

The logic model's main limitation is that it may lead to over-simplification. Medical educational programs are often complex and don't always follow a linear path, and outcomes may not match what is initially predicted. To overcome this limitation, the logic model needs to be well designed. The creators of the model should have a thorough understanding of how change works in the educational program being evaluated. Both intended and unintended outcomes should be anticipated, and feedback loops must be incorporated into the model to address these complexities.

Complexity can be further built into the Logic Model with the addition of multiple tiers, accounting for various layers of an intervention. Mills et al attempt to address this problem in their 2019 article. [] The authors propose a typology of logic models. They categorize logic models into four types, ranging from simple (type 1) to the most complex (type 4). The type 4 logic models attempt to provide more insight into the interactions between interventions and context (social, political or cultural factors in the environment where the program exists).

The logic model design needs to be flexible and dynamic to integrate unexpected complexities. Educators and researchers need to be prepared to revise the model as the program is being implemented. Therefore, the development and revision of a logic model can be a time-consuming process.

Finding the right balance between precision (which may require many data points) and clarity (emphasizing concise, easy to understand points) in your logic model can be a significant challenge, especially when getting started.

Part 3: The Denouement
Sarah uses the logic model to plan a curriculum for her coworkers to obtain core ultrasound skills in a stepwise approach. She creates a pre-survey and pre-test to see initial attitudes and knowledge. She then creates educational opportunities to teach her coworkers. She has a post-survey and post-test to evaluate how behaviors, knowledge, and attitudes have changed over the course of a year, as well as how the number of ultrasounds performed in the department during clinical practice has changed. Sarah makes a goal of certifying 75% of her colleagues in core ultrasound applications at first, and then plans to revisit her logic model and expand her program to include other imaging applications. The logic model she produced is here.

Don't miss the eighth post in the series, coming out Tuesday, September 21, 2021!
PLEASE ADD YOUR PEER REVIEW IN THE COMMENTS SECTION BELOW

References

1. McLaughlin JA. and GB Jordan, Logic models: a tool for telling your programs performance story . Evaluation and Program Planning, 1999. 22(1): p. 65-72.

2. Developing a Logic Model or Theory of Change. [cited 2020 May 22]; Available from: https://ctb.ku.edu/en/table-of-contents/overview/models-for-community-health-and-development/logic-model-development/main.

3. Otto AK, K Novielli, and PS Morahan. Implementing the logic model for measuring the value of faculty affairs activities. Acad Med, 2006. 81(3): p. 280-5.

5. Schum an E. Evaluative Research Principles and Practice in Public Service and Social Action Progr . 1968, New York: Russel Sage Foundation

6. Bickman L. The functions of program theory. New Directions for Program Evaluation, 2004. 1987: p. 5-18.

7. Wholey, J., Evaluation and Effective Public Management. . 1983, Little Brown.

8. Measuring Program Outcomes: A Practical Approach ed. U.W.o. America. 1996.

10. Glynn MK et al. Strategic Development of the Public Health Workforce: A Unified Logic Model for a Multifaceted Program at the Centers for Disease Control and Prevention Journal of public health management and practice : JPHMP, 2019.

13. Railer J et al., U sing outcome harvesting: Assessing the efficacy of CBME implementation. J Eval Clin Pract, 2020.

16. Van Melle E. Program Evaluation for CBME: Are we making a difference? in The International Conference on Residency Education. 2016. Niagara Falls, Ontario, Canada.
The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our 'About' page

# data[].attributes.attachment_link.title
Education Theory Made Practical - Volume 5, Part 7: Logic Model of Program Evaluation

# data[].attributes.attachment_link.description
The Academic Life in Emergency Medicine (ALiEM) Faculty Incubator was hard at work during the pandemic to bring you the fifth volume of the Education Theory Made Practical series. This series strives to make theory accessible to educators by distilling the background and key literature of each theory and grounding them in practical education scenarios....

# data[].attributes.headline
HE specific review standards from QM for online course evaluation

# data[].attributes.subject
HE specific review standards from QM for online course evaluation

# data[].attributes.attachment_link.readable_text
The Rainbow Framework can help you plan an M&E activity by prompting you to think about each of these tasks in turn, and select a combination of methods and processes (which we call 'options') that cover all tasks involved. You might also choose an approach, which is a pre-packaged combination of options.

Download the Rainbow Framework

There are two downloadable versions of the Rainbow Framework - one only shows the tasks, and one also includes all the options (methods and processes).

View downloads

Explore the Rainbow Framework

The Rainbow Framework shows the different options (methods or processes) that can be used for each task in an evaluation. Most of these tasks are needed in any M&E activity, but some (marked with an asterisk) might not be needed, depending on the purpose of the M&E activity or the specific type of evaluation you aim to conduct.

MANAGE an evaluation or M&E system

Decide how the evaluation or the M&E system will be managed, including clarifying stakeholders, roles and decision making processes, and ensure processes for these are transparent and well-managed.

Click to view evaluation tasks for MANAGE
Evaluation tasks:

1. Develop initial description
Develop an initial description of what is to be evaluated, to ensure agreement about the boundaries.

2. Develop programme theory / theory of change *(optional)
Make explicit how activities are understood to contribute to the intended outcomes and impacts. Communicate this in a diagram(s) and/or narrative. (This is useful for most evaluations but not always needed, such as for evaluations of the quality of products).

3. Identify potential unintended results
It is useful and ethical to consider possible negative impacts (that make things worse not better) and how they can be identified before an intervention (project, programme, or policy) is implemented and addressed in an evaluation or M&E System.

Evaluation tasks:
These tasks are particularly relevant for outcome or impact evaluation. The tasks reflect three broad strategies that can be used for understanding what caused the observed outcomes or impacts. While each strategy can stand alone, using a combination can usually help to increase the strength of the conclusions drawn.

1. Check the results are consistent with causal contribution (*optional)
Check that the data are consistent with what would be expected if the intervention were contributing to producing the observed changes.

2. Compare results to the counterfactual (*optional)
Develop an estimate of what would have happened without the intervention and compare that to the findings of what happened with the intervention.

3. Investigate possible alternative explanations (*optional)
Identify other factors that might have caused the impacts and see if it is possible to rule them out.

Evaluation tasks:

1. Synthesise data from a single evaluation
Decide how data will be combined in terms of the agreed evaluative criteria and standards to produce an overall judgement of merit or worth.

2. Synthesise data across evaluations (*optional)
Decide how to find, extract and combine data from multiple evaluations to produce more general conclusions about 'what works' or 'what works for whom in what circumstances'. (This is not needed for a single evaluation).

3. Extrapolate findings (*optional)
Explain how findings from this evaluation might be more generally applied or translated to new sites and situations. (This is often useful but not always needed).

# data[].attributes.attachment_link.description
The Rainbow Framework can help you plan an M&E activity by prompting you to think about each of these tasks in turn, and select a combination of methods and processes (which we call 'options') that cover all tasks involved. You might also choose an approach, which is a pre-packaged combination of options.

# data[].attributes.headline
Centre for Education Statistics and Evaluation - NSW, Australia

# data[].attributes.subject
Centre for Education Statistics and Evaluation - NSW, Australia

# data[].attributes.attachment_link.description
Empowering schools to strengthen their practice.

# data[].attributes.headline
Intelligent system to measure the quality of assessments in HE

# data[].attributes.subject
Intelligent system to measure the quality of assessments in HE

# data[].attributes.attachment_link.readable_text
The quality criteria presented in the form of a checklist are intended as assistance to authors and users of open educational resources and the organisations behind them. The quality criteria will:

help authors create resources of a higher quality from the perspective of openness;

help users in quality assessment as they are reviewing open educational resources;

give the background organisations a tool they can use for facilitating the preparation and evaluation of open educational resources as part of other guidelines and instructions provided by the organisation.

The guidelines define five different quality criteria for open educational resources and more detailed quality dimensions for each of them. The criteria are:

Content. The content of high-quality open educational resources is reliable.

Learning and teaching methods. High-quality open educational resources enable the use of diverse ways to learn and teach.

Usability and accessibility. High-quality open educational resources are as easy to use as possible for everyone and in any situation.

Findability and sharing. High-quality open educational resources are easy to find, and the users know under which terms and conditions they can share them.

Lifecycle and adaptability. High-quality open educational resources are maintained and updated on a regular basis. In addition, other open resources and co-development practices are used in the production of the materials.

Under each criterion, the quality dimensions have been arranged according to different levels - a check in the box indicates that the element applies to open educational resources of this level. The first-level quality dimensions describe basic characteristics that all open educational resources should have. The second-level quality dimensions describe more advanced open educational resources in terms of quality and the third-level quality dimensions apply to highly advanced open educational resources.

The guidelines are based on the international TIPS Quality Assurance Framework for open educational resources. The guidelines focus on quality dimensions of particular importance for openness. General instructions on the quality of online implementations and related materials can be found, for example, in eAMK Quality criteria for online implementations.

Dimensions of the content criterion

Dimensions of the learning and teaching methods criterion

Dimensions of the usability and accessibility criterion

Dimensions of the findability and sharing criterion

Dimensions of the lifecycle and adaptibility criterion

# data[].attributes.attachment_link.description
The quality criteria presented in the form of a checklist are intended as assistance to authors and users of open educational resources and the organisations behind them.

# data[].attributes.attachment_link.readable_text
Ok, so now you're leaning towards adopting Open Education Resources, but are wary because you're concerned about the quality of the resources. They can't be that good if they're free, you say? Here's what others think about OER, and why you shouldn't be scared off by the lack of a price tag!

# data[].attributes.attachment_link.title
LibGuides: Open Source Education Resources: OER Quality Assurance

# data[].attributes.attachment_link.description
Ok, so now you're leaning towards adopting Open Education Resources, but are wary because you're concerned about the quality of the resources. They can't be that good if they're free, you say? Here's what others think about OER, and why you shouldn't be scared off by the lack of a price tag!

# data[].attributes.headline
Evolution of quality assurancepractices in enhancing the qualityof open and distance educationin a developing nation: a case study

# data[].attributes.subject
Evolution of quality assurancepractices in enhancing the qualityof open and distance educationin a developing nation: a case study

# data[].attributes.attachment_link.description
Welcome to TECH4101 E-Portfolio Done by: Ibrahim Alabri 125620

# data[].attributes.attachment_link.readable_text
Evaluation in Educational Technology - TECH4102

This is where all my work in the course of Evaluation in Educational Technology (TECH4102) will be.

I hope you'll enjoy viewing my work. Please go ahead and explore, and don't hesitate to contact me or interact if you have any comments :D!

​

Name: Ashwaq Al Yafai , ID: 126295

Instructor: Raja Maznah

# data[].attributes.attachment_link.description
Evaluation in Educational Technology - TECH4102 This is where all my work in the course of Evaluation in Educational Technology (TECH4102) will be. I hope you'll enjoy viewing my work. Please go ahead and explore, and don't hesitate to contact me or interact if you have any comments :D!

# data[].attributes.body
ePorfolio

# data[].attributes.attachment_link.readable_text
Welcome to my e-portfolio. My name is Marwa Al Mahrouqi.I have four years experience as IT teacher. I graduated from Sultan Qaboos University, collage of Education, Instructional and Learning Technologies. I am doing my master on the same major. Wish me all the best.

# data[].attributes.attachment_link.description
Welcome to my e-portfolio. My name is Marwa Al Mahrouqi.I have four years experience as IT teacher. I graduated from Sultan Qaboos University, collage of Education, Instructional and Learning Technologies. I am doing my master on the same major. Wish me all the best.

# data[].attributes.body
ibrahim-alalawi-portfolio

# data[].attributes.attachment_link.description
As an ambitious teacher, I have decided to sail in the deep inspirational world of technology due to its significance in improving my noble profession... So, welcome to all people sailing with me in the same boat of my modest e-portofolio. I

# data[].attributes.attachment_link.readable_text
Abstract

The growing shift toward online learning has brought new expectations for teachers, including skills needed to combine content knowledge with engaging pedagogical strategies that leverage the affordances of technology. As a result, online pedagogy has become increasingly relevant in modern-day schools. The challenge is understanding the nature of online pedagogy, the skills needed for teachers to succeed in online settings, and the theoretical underpinnings surrounding why these skills are essential. This article unpacks the foundational components of online pedagogy, comprised of five pillars grounded in the principles of learner-centeredness, constructivism, and situated learning. These pillars include the ability to (a) Build Relationships and Community, (b) Incorporate Active Learning, (c) Leverage Learner Agency, (d) Embrace Mastery Learning, and (e) Personalize the Learning Process. We describe their theoretical underpinnings, discuss related literature, and consider implications for teacher education with subsequent implications for scholarship across educational technology, educational psychology, and the learning sciences.

# data[].attributes.attachment_link.title
Pillars of online pedagogy: A framework for teaching in online learning environments

# data[].attributes.attachment_link.description
Abstract The growing shift toward online learning has brought new expectations for teachers, including skills needed to combine content knowledge with engaging pedagogical strategies that leverage the affordances of technology. As a result, online pedagogy has become increasingly relevant in modern-day schools.

# data[].attributes.headline
Threshold standard for online learning environment

# data[].attributes.subject
Threshold standard for online learning environment

# data[].attributes.headline
Abdulsattar_The quality of selected online ...

# data[].attributes.subject
Abdulsattar_The quality of selected online ...

# data[].attributes.attachment_link.readable_text
The COVID-19 pandemic has led to the largest interruption in education around the world, affecting some 1.6 billion students. Classroom education stopped, to be replaced by online e-learning platforms. In higher education, e-learning is made available through recorded lectures, with online platforms becoming a significant part of the overall system. This study aims to identify the quality of online learning platforms, applying a set of criteria from the perspective of faculty members and students in higher education institutions in the Sultanate of Oman. The descriptive approach is used and the researchers collected data through a questionnaire directed at 32 faculty members and 104 students. The results show that e-learning programs are generally of high quality, from the perspective of the participants, but with statistically significant differences according to the type of program.

1. Introduction

E-Learning is a successful product in the current era of technological innovation, and its introduction has become inevitable in light of the current and accelerating changes in all developed and developing societies. Zaghdoud [ 1] comments that the Internet and multimedia technologies have reshaped the way knowledge is presented, as e-learning has become an alternative to traditional education.

Most of the literature indicates that e-learning is characterized by effectiveness, one of its advantages overtraditional methods; it is less expensive as it saves learners the cost of travel, earning them more time and providing not only educational resources but also assessment tools that allow them to measure their progress. It is also characterized by flexibility as it allows the learner to find information from anywhere and at an appropriate time. It contributes to creating an environment that increases opportunities for cooperative education, thus transferring learning to a more realistic environment, away from the artificial situation that isolates learners from each other.

E-Learning platforms are an integrated set of interactive services that are not restricted by time or place and are provided through the Internet, with the necessary tools and materials for teachers, learners, and those concerned with learning. They support and enhance the educational process, through which learners can access courses and participate in various learning activities. They help to create collective spaces for learners, enabling them to communicate with each other from a distance, exchange information, and solve problems [ 13].

Fathi and colleagues [ 17] emphasized that the success of any educational and training system depends largely on its commitment to internationally agreed quality standards, as education is not provided in a random way but rather is a well-planned and designed system. In order to achieve and ensure quality in e-learning, three basic conditions must be met: ensuring real growth in the learner's personality and behavior; alignment with the needs of the community under the existing conditions, given the practical and professional characteristics of the institution; and the availability and application of accurate standards and indicators for the program's design and production. Many studies have confirmed that if e-learning is not designed well, taking into account quality standards and indicators, it will not add any value to the educational process.

The process of achieving quality in e-learning faces many challenges, the most important of which is the compatibility of the quality of the educational service provided to the beneficiary in the e-learning system with the level of service quality that he expects [ 8]. Different definitions of quality involve their close association with standards, the gateway to achieving quality in education in any institution. Muhammad and Abdulnaeem, referred to in [ 8], defined quality as an endorsement of a set of items that prove by scientific study and careful research that they are sufficient to meet the critical needs to complete the product in its final form. It determines the specifications and conditions that should be available in the education system [ 8]. Harthy, cited in [ 18], defined quality in education as a comprehensive system that deals with the various aspects of the educational system through inputs, processes, and outputs, with the aim of improving its products.

Qudah [ 19] adds that one of the most important reasons for academic institutions' interest in the quality standards of the educational process is to raise and improve the level of their outputs. This is confirmed by the standard specification, the most important of which is the focus on the service to recipients, which calls on universities to adopt clear methods to identify the degree of students' satisfaction with the services provided by the educational institution, and the performance and level of response of universities to the requirements and needs of students.

In order to achieve quality in e-learning, a set of conditions must be met: students enrolled in this type of education should have appropriate input capabilities; this type of education should be related to the educational content and the applications or programs used; it should include faculty development and continuous evaluation; it requires close monitoring of the implementation of instructional programs.

2. Research Problem

Most of the higher education institutions in the Sultanate of Oman turned to e-learning during the COVID-19 pandemic, as an emergency alternative to ensure the continuity of education and learning during periods of home quarantine; the use of instant messaging applications and platforms increased rapidly [ 20]. The aim was primarily to reach students and teach them in their homes, providing them with educational materials and activities. Müezzin [ 12] found that after private and public educational institutions were closed, many of them resorted to video chat programs via the Internet, including various Google programs and GoToMeeting. Despite the existence of these alternatives, the educational process still experienced many problems. These included teachers and students being unfamiliar with this type of education, and the fact that not all students had access to the Internet and electronic devices.

Alameri et al. [ 21] added that a goal of these platforms was that university students and faculty members in technical and humanitarian disciplines should learn innovative technologies, as a theoretical and practical foundation for skills in the field of innovation in education. The absence of face-to-face communication and the new e-learning environment posed challenges for both learners and teachers, increasing their workloads, especially in the time and effort expended in preparing content and in learning activities. While the quality of the learning services provided through these platforms could not be relied on, the quality of individual teaching and learning might also be reduced [ 22].

Latipet al. [ 22] identified difficulties in achieving quality standards for e-learning in Arab countries, given that the e-learning systems may have been designed for different cultures and environments. On one hand, they failed to involve learners in activities to ensure good quality and, on the other, to identify good practices in this field of e-learning. There were no formal procedures or policies to review the quality of the e-learning system, develop it, and monitor its effectiveness. Evaluation methods that comply with quality standards must, therefore, be approved and adopted. In addition, users must be aware that the provision of content through e-learning platforms requires its own standards [ 3]. Afifi and Al-Omari [ 10] believe that the majority of quality models in e-learning and distance education focus on standards for designing only the educational content, without considering the other elements of the e-learning system. In particular, they recommended the need to pay attention to developing e-learning quality standards in light of the growing competition between universities in providing applications and services for this type of education. Otaibi and Roqi [ 8] added that the problem also arises whenever they are subject to the standards of nonprofit organizations. Hayat [ 18] confirms that the educational programs adopted in the e-learning system require continuous evaluation and feedback in order to be developed and reformed. Quality in e-learning is a prerequisite for the success of the educational process, as it largely depends on commitment to globally agreed standards in both the technology used and the e-learning programs themselves [ 23].

Umek and colleagues [ 24] confirm that it is important to analyze the opinions of the participants and stakeholders about the use of e-learning platforms, with the availability and application of accurate standards for their design and production. It must be remembered that e-learning is not just another way of applying traditional teaching, but rather a new approach to education, and that quality assurance methods must take this fact into account [ 25].

On the other hand, [ 26] believed that it is important to recognize traditional cultural, social, and educational values when adopting general evaluation models and criteria for e-learning and to adapt these models to suit the community and therefore ensure the quality of e-learning.

Given the importance of improving e-learning in the Sultanate and the proximity and nature of the researchers' work, and sensing the importance of evaluating the experience of the e-learning process in higher education institutions and educational platforms in accordance with international quality standards, in a manner commensurate with the nature of Omani society, this study asks "What is the reality of the quality of e-learning platforms used in higher education institutions from the perspective of the Omani user?"

The following emerged from this question:(i) What is the reality of the quality of e-learning platforms used in Omani higher education institutions from the perspective of faculty members and students? (ii) Are there statistically significant differences (at the 0.05 level) in the quality of e-learning according to the different programs used, from the perspective of students and faculty members? (iii) Are there statistically significant differences in the quality of e-learning platforms between Omani public and private higher education institutions, from the perspective of faculty members? (iv) Are there statistically significant differences in the quality of e-learning platforms between Omani public and private higher education institutions from the students' perspective?

The significance of the study from the theoretical perspective is that it is the first of its kind, to the researchers' knowledge, in the field of evaluating e-learning platforms based on approved international standards in the Sultanate of Oman. From a practical perspective, it will advise those in charge of higher education on the quality of the e-learning platforms used. The answers to the research questions will contribute to raising the quality of e-learning in Oman.

Finally, the limitations of the study are as follows:(i) Spatial boundaries: this study is limited to higher education institutions in the Sultanate of Oman (ii) Temporal limits: this study was conducted in the second semester of the academic year 2020/2021 (iii) Objective limits: the study is limited to evaluating the e-learning platforms Microsoft Teams, Google Meet, and Zoom (iv) Human limits: this study is limited to faculty members and students in Omani higher education institutions

3. Literature Review

3.1. Overview

E-Learning via the Internet, mobile devices, and social networking tools has a contemporary constant replacing traditional classes. The process of shifting from the traditional system of education to the electronic system is a complex process that requires preparation and the provision of the appropriate tools and supplies to meet educational goals. It must take into account not only the abilities and skills of students but also their culture and privacy when communicating and interacting [ 27].

A United Nations report [ 28] indicated that the COVID-19 pandemic caused the largest interruption to education in history, affecting about 1.6 billion students. As there are currently no indications that this crisis and the unprecedented interruption to education will end, online learning has been made available for higher education through recorded lectures and online platforms. It is, therefore, important to seize this opportunity to find new ways to confront the learning crisis and develop a set of sustainable solutions, including digital solutions that require institutions to have appropriate content, effective teaching models and practices, and an enabling learning environment. There is a report [ 29] that e-learning has thus imposed itself strongly on academic institutions, commensurate with modern technological developments, and that the introduction of modern technology in the field of higher education is one of the main pillars of quality in education.

Hayat [ 18] summarized the importance of e-learning as contributing to facilitating and increasing communication and exchanging opinions between students and their teachers. e-Learning tools, including e-mail, dialogue, chat, and instant forums, help to form knowledge and strong opinions among learners. On one hand, courses can be constructed in light of specific scientific standards; on the other hand, they enable students to learn individually and according to their own abilities and encourage them to manage their learning in the manner that suits them. In addition, e-learning provides all the courses and documents needed and references related to topics being studied.

E-Learning facilitates the process of education outside the working hours of educational institutions and thus helps students to acquire self-learning skills and reduce time and effort. It also encourages teachers and learners to use modern technologies and raise the level of communication between them, in addition to the possibility of providing educational materials in different forms [ 30].

E-Learning has many advantages for students, including the ability to accommodate large numbers of learners and take into account individual differences between them, as they can complete learning processes in environments appropriate for them and progress according to their own abilities. It also increases learners' sense of justice and equality in the distribution of opportunities in the educational process. From expressing their ideas to searching for information and facts by different means in addition to traditional methods [ 19, 20], e-learning ends the learner's isolation and keeps him in constant contact with his educational institution. Educational content is updated with ease, but it is necessary that the programs and applications of e-learning services are of high quality.

Al-Qudah and Hamed [ 19] stated that it is important to recognize the quality of the education process, not as something that is delivered to the learner by the e-learning provider but rather as a coproduction between the learner and the learning environment. The e-learning used may be synchronous, bringing together all the learners at the same time, with direct communication established between them so that they can interact directly with the teacher. Examples of such tools are video conferencing and virtual classes. Alternatively, asynchronous e-learning does not require the presence of all the students at the same time; each can obtain instructional materials according to a preplanned program, choosing the appropriate time and place to study, with tools including e-mail, video media, and CDs [ 31].

Providing quality in e-learning is essential for any academic program or course, a prerequisite for the success of the educational process. Therefore, adherence to agreed international quality standards is required to guarantee the success of any educational system; these standards act as a social contract between teachers and education authorities on the one hand and between the education authorities and students on the other hand [ 18]. Otaibi and Roqi [ 8] stressed that applying total quality standards in educational institutions leads to qualitative development, in line with educational and administrative developments to achieve excellence in all their services.

The importance of e-learning quality standards lies in the fact that a number of criteria must be met to help students make the right choice among the many learning opportunities available to them, which vary in terms of quality and availability. Equally, they are needed by educational institutions and universities to meet the needs of students and provide efficient and effective services. This reduces the conflict between what is taught and what must be achieved and also helps in judging the quality of learning [ 18].

The criteria for accreditation and quality assurance vary from one country to another, depending on the objectives of the e-learning provided and also according to the sociocultural conditions of each country. Wenceslao [ 32] stated that most of the studies reviewed identified the following criteria for evaluating e-learning: ease of use, response time, interaction, web design and courses, accessibility, reliability, cost-effectiveness, functionality, security, stability, accuracy, flexibility, interoperability, and continuity. Many institutions and organizations have sought to set special standards to ensure the quality of e-learning, each party according to its perspective and interest. They include the Association of Asian Open Universities, which has developed a framework to ensure the quality of e-learning, with eight basic areas for professional development processes. The Chinese Committee for e-learning Technology Standards, commissioned by the Chinese Ministry of Education, has identified four directions for deriving quality standards for e-courses. In the same field, the European Association of Distance Learning Universities has prepared standards for the quality of e-learning systems and has developed a guide containing a set of standards and indicators for evaluating the quality of e-learning in general [ 33].

Canadian standards for e-learning consist of three main areas. The first is the quality of inputs for e-learning products and services, with thirteen dimensions. The second area, on which this study focuses, is the quality of processes and practices in e-learning services and products. It has five dimensions, the first four coming under the heading of communication management: managing and supporting students, connecting and managing learning, building knowledge, and improving students' ability to solve problems. The fifth dimension, privacy and personal management, comprises achievement files that allow evaluation of the learner and the provision of a long-term inventory characterized by privacy, security, and ease of access. The third area is the quality outputs of e-learning services and products, with three basic dimensions [ 34].

The challenges to the effectiveness of e-learning are summarized by [ 8] lack of compatibility between the culture of e-learning elements and the evaluation requirements according to quality standards of the teacher, the learner, and the technical and administrative staff; inadequate quality of the educational services provided in the electronic system, compared with the level of service required by the teacher and the learner; the inadequacy of existing conditions in the e-learning system according to quality standards; and the desire for immediate, not long-term, results.

3.2. Related Works

Ngoc and Phung [ 7] investigated the challenges facing students learning English as a foreign language online through Moodle and Microsoft Teams platforms. A questionnaire and semistructured interviews were used to collect study data, and the results showed that students had negative attitudes toward many aspects of online education, as they faced technical problems and lack of social interaction.

Al Mulla and Abdullah [ 13] evaluated the digital platforms used in distance education in international schools in Kuwait, from the perspective of art education teachers and directors, using a questionnaire to collect study data from a sample of 45 teachers and 11 directors. The study concluded that the best standards in terms of availability and use were the digital content standards on the platform and the digital content display standard.

Wea and Kuki [ 35] examined the perceptions of students of the College of Teacher Education and Training of the University of Nusa Nipa in Indonesia about the use of the Microsoft Tamayuz application. Data was collected by questionnaire from the study sample of 176 students, and the results showed that the students had positive perceptions about the use of the e-learning program.

Latip and colleagues [ 21] examined the effect of using Moodle, Microsoft Discrimination, and Zoom platforms on self-learning and academic performance, with a sample of 450 students. The advantages and disadvantages of e-learning were included in the quality content analysis, and the main result was that the strategic design of the university's e-learning program was more important than individual contextual variables in assessing students' perceptions. Students considered that the e-learning communication on the campus was very effective and that those who had an experience of using the new technologies for learning had positive attitudes toward e-learning.

Rizki [ 30] analyzed the perception of economics students at the University of Bordj Bou Arreridj, Algeria, about the reality of e-learning and its obstacles. The study sample consisted of 42 male and female students, with a questionnaire to collect data. The students acknowledged the importance of e-learning and did not agree that there were obstacles.

Ibrahim and Al-Nafi'i [ 36] investigated the standards of teachers in the International Society for Technology in the field of education in Oman and recommended the development of technology standards for teachers based on the ITA Standards for Technology in Education.

Bin [ 27] evaluated the experience of e-learning in a course on the skills in searching for information. The researcher used a questionnaire to collect students' opinions about their experience of the electronic course. The results showed the importance of e-learning in terms of facilitating students' study and following up on their development and activity during the semester.

Al-Qudah and Hamed [ 19] evaluated the quality of e-learning and its impact on the degree of satisfaction of students at Taibah University, Saudi Arabia, using a questionnaire distributed to a random sample of 300 male and female students. The study concluded that the general trend of satisfaction was high. It recommended that the university continues to adopt e-learning as a strategic option and to learn about the best international practices in the field of e-learning.

Khdeem's study [ 3] aimed to diagnose the reality of distance education platforms in higher education institutions in Algeria, using some website analysis tools to collect data. The results showed a shortage of distance e-learning support: installation, updating, insurance, and response to various screens. Traditional methods of education should be complemented by e-learning.

Mahnawy and Ahmed [ 37] studied how to employ e-learning to achieve quality standards in secondary education in Iraq. The study sample from 15 schools comprised 800 male and female students and 20 computer teachers. A questionnaire was used to collect data, and the study concluded that it was necessary to employ e-learning to improve secondary education.

Alameri and colleagues [ 21] investigated students' opinions about the effectiveness of e-learning using digital learning platforms Moodle, Microsoft Teams, and Zoom and the impact on their self-learning and academic achievement. The sample consisted of 450 male and female students. The researchers pointed to many results, including that most students believe that the three platforms are important for all universities today, helping their learning activities more effectively than traditional methods, improving their self-study abilities, and encouraging them to progress in line with their abilities. The results also showed that the quality of Moodle, Microsoft, and Zoom curricula reflected the educational level of universities at the present time.

Zahia [ 31] aimed to identify the role of the two types of e-learning, synchronous and asynchronous, during COVID-19. The researcher used the ethnographic method, studying data and reports from the University of Hail in Saudi Arabia. The results showed that the university was adopting a comprehensive approach to the digital environment and overcoming all obstacles faced by teachers and learners through field training, facilitating the process of virtual education, and confirming the effective role played by e-learning in the success of the university year.

The study by [ 38] aimed to identify the challenges and problems facing students of higher institutions related to the transition to e-learning during the COVID-19 pandemic, with a sample of 141 male and female students. Results indicated that the most important barriers were their unwillingness to study online, social challenges, and the lecturer's challenges. Thakker et al. [ 39] investigated the perceptions of engineering students about the available e-learning platforms, with the aim of providing solutions to help enhance e-learning in the long term. The study sample of 364 students indicated that Google was the preferred system.

Karima [ 2] investigated the role of e-learning within educational institutions and research centers, using the descriptive-analytical method, a sample of 150 members of the university faculty, and a questionnaire to collect data. e-Learning achieved good results in educational service quality, the needs of teachers, and the educational process.

Otaibi and Roqi [ 8] aimed to identify the concept of quality, its principles, and its importance in the educational process, as well as reviewing the quality standards adopted for the e-learning system. These standards varied in terms of their number, the diversity of studies, and the institutions involved.

Finally, Afifi and Al-Omari [ 10] proposed a model for measuring aspects of quality and standards in e-learning and distance education programs at the University of Dammam, Saudi Arabia.

4. Data Collection and Analysis

The researchers applied the descriptive-analytical approach to achieve the objectives of the study, as it describes a certain phenomenon and reports its status as it is in reality [ 40], by reviewing the literature, building measurement tools, and collecting data from the study community. The analysis provides answers to the research questions and suggests recommendations. The study community consisted of faculty members and students in Omani higher education institutions. A questionnaire was distributed electronically because of the COVID-19 conditions. The sample consisted of 124 male and female students and 32 faculty members from 24 public and private universities in the Sultanate.

4.1. Data Collection Tool

The researchers designed two scales to measure the study data, collected by a questionnaire on the quality of the programs used in e-learning, one directed to faculty members and another to students. The Canadian standards for e-learning were adopted, containing the five dimensions already described, presented in Table 1 [ 34]. The researchers added items on the quality of the programs used in e-learning, with two open questions for students to offer an opinion of the challenges facing them and for faculty members to obtain their suggestions for improving the quality of e-learning programs.

A five-point Likert scale was used to measure the responses to the closed-answer items, as detailed in Table 2.

The researchers then checked the validity of the study tools, by presenting the originals to a group of arbitrators. The questionnaire items were modified in accordance with the arbitrators' recommendations on their content and presentation.

The consistency of the electronic form of the questionnaires was tested with an exploratory sample of the study population, and Cronbach's alpha was calculated. Using the SPSS program, the results showed high stability coefficients, approaching 1, where the resolution for faculty members reached 0.92 and for students 0.98; see Table 3.

4.2. Procedure

The researchers reviewed the literature in order to define the problem accurately and on a clear scientific basis and to clarify the importance of addressing the study questions and objectives theoretically and scientifically.

They then identified the study community and transferred the study questionnaires into an electronic form as described above. The questionnaires first explained the objective of the study and gave instructions for answering the questions. Respondents were then required to provide demographic information and the type and name of the educational institution to which they belonged. The final section consisted of items in the five dimensions to measure the quality of e-learning programs. The design, validity, and reliability were confirmed using SPSS.

After the validity of the measurements and their relevance to the objectives of the research had been assured, the researchers published the electronic questionnaire on various social networking sites, most importantly WhatsApp and Twitter, with the aim of reaching the widest possible study sample from various educational institutions. The data was then entered into SSPS and checked as suitable for statistical treatment. After analysis, the results were evaluated in relation to the research questions and compared with the results of previous studies. Finally, the results were summarized and recommendations made.

4.3. Results and Discussion

In order to interpret and analyze the results, the researchers reduced the Likert measures to a triple scale to facilitate the interpretation of the average response; see Table 4.

By type of institution, around two-thirds of faculty members and students were from public academic institutions and the remainder from private ones. The detailed figures for staff and students are presented in Tables 5 and 6, respectively.

The analysis of the measurement of responses to items in the questionnaire is as follows.

The first question was "What is the reality of the quality of the e-learning platforms used in Omani higher education institutions from the perspective of faculty members and students?"

Most of the measures of quality as perceived by students (Table 8) were high, although the dimension of privacy and assessment management was only medium (3.2 ± 1.1).

When faculty were asked about the most important challenges to using e-learning platforms, their responses were limited to technical issues, weak Internet connection, inability to organize time, lack of a mechanism for monitoring and evaluating students, checking cheating operations, the process of registering and adding students, maintaining students' focus, and interaction during the learning process.

Students' responses to the most important challenges in using e-learning platforms were technical issues, weak Internet connection, difficulty studying practical materials through e-learning platforms, difficulty adapting to the new system of education, faculty members not having technical skills in education, the absence of real interaction between the teacher and students, inability to focus, feelings of boredom, and inability to organize their time.

Concerning suggestions for developing and improving the quality of e-learning through electronic platforms, the responses of faculty members included the following proposals: conducting training workshops to promote the use of electronic platforms, with a manual for their use, a mechanism to enlist students collectively rather than individually, support with devices for presenting lectures such as a drawing pad, changing the culture of society, accepting the use of cameras, creating an appropriate environment for students to receive the lessons; reducing Internet subscription prices for students and faculty members, working on a specialized e-learning platform for each institution, finding a mechanism to update devices used in e-learning, employing technicians to help, and providing more tools to explain and clarify the system and to facilitate the communication process.

The students also had many proposals: finding a solution to the poor Internet provision and making it free to students, continuing e-learning for some subjects, improving the efficiency and skills of faculty members in educational advances via the Internet, each institution adopting a unified platform for all its students, finding a way to motivate students to communicate and interact through electronic platforms, providing materials and explanations to help students' learning, and providing devices for students.

The second question was "Are there statistically significant differences ( p  = 0.05) in the quality of e-learning in the programs used in e-learning, from the perspectives of students and faculty members?"

ANOVA was used to analyze the answers to this question, with the results as shown in Table 9 for faculty members and Table 10 for students. Overall, there are no statistically significant differences in the quality of e-learning by type of program.

The third question was "Are there statistically significant differences (0.05) in the quality of e-learning platforms between Omani public and private higher education institutions, from the perspective of faculty members?"

The t-test of independent samples was used to calculate statistical differences in the responses by type of academic institution, with the adoption of the 0.05 significance level. The results in Table 11 indicate no statistically significant differences ( in the quality of educational platforms by type of institution in four dimensions: student management, learning management, communication management, and privacy and assessment management, respectively, at 0.2, 1.8, 1.9, and 0.4 at significance levels of 0.8, 0.07, 0.06, and 0.7. However, there was a statistically significant difference at the 0.05 level for the ease of use dimension, in favor of private education institutions ( t  = 2.8) with a significance level of 0.02.

The fourth question repeated the third, but from the students' perspective.

The same procedure was followed, and the analysis, as shown in Table 12, found no statistically significant differences at the significance level of 0.05 in the quality of educational platforms by type of academic institution in four dimensions: learning management, communication management, ease of use, and privacy and assessment management, at 2.1, 1.8, 0.4, and 0.9 with significance levels, respectively, of 0.54, 0.68, 0.69, and 0.38. However, a statistically significant difference at the 0.05 level was found for the dimension of student management, in favor of public education institutions ( t  = 2.1 and

The answers to the first and second questions indicate that the programs used in e-learning are of high quality. The researchers attribute this result to the fact that all these programs were designed and created by the largest technology companies in the world.

Regarding challenges related to e-learning, all respondents believed that they might affect the quality of education, through the problems listed above. The researchers conclude that it is necessary to provide strong support mechanisms for both faculty members and students, making continuous evaluations of e-learning programs and applications, enhancing the e-learning culture, and cooperating with stakeholders to find mechanisms to ensure continuous improvement through a strong technological infrastructure, drawing on local and global expertise to improve and upgrade e-learning.

The results for the third and fourth questions showed statistically significant differences by institution type, but only for the dimension of ease of use in favor of private education institutions in the case of faculty members, and the dimension of student management in favor of public education institutions from the students' perspective.

5. Conclusion

E-Learning platforms were originally established to support the education process, but during the COVID-19 pandemic, they have played a major role in ensuring the continuation of education. Microsoft, Google, and other high-tech companies have designed and implemented useful e-learning platforms for all education levels, from primary school to university. This study aimed to validate the previous facts about e-learning and show that the programs used in the Sultanate enjoy high quality, from the perspective of faculty members and students, with no significant problems related to quality in general.

The analysis showed that faculty members and students do face some challenges related to Internet connection services, poor technical support, inability to organize learning times, the lack of a mechanism through which the faculty member can enroll students in the program collectively, the difficulty of maintaining focus, the lack of mechanisms to verify the absence of cheating, and the weak skills of faculty members and students in dealing with e-learning tools. Many of these problems arise from the fact that the systems were developed in an emergency situation with no or limited prior experience.

Indeed, the current reality indicates that the near future will see more developments in the field of e-learning because of its active role in the continuation of the education process in various parts of the world. These developments will focus on how to reduce the difficulties identified and increase the efficiency of e-learning.

Data Availability

This article aims to support researchers who are interested in the field of education, by sharing the research data and the template of the data collection tool through the following link: https://drive.google.com/drive/folders/1THMhsPBEtwdr7yTReXhhXeYPzBC2-mrc?usp=sharing.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

The authors gratefully acknowledge the financial support of the Ministry of Higher Education, Research, and Innovation in Oman, under Grant no. MoHERI/BFP/MC/01/2020. They also acknowledge the infrastructure and support of Muscat College. Finally, the authors would like to extend their thanks to the faculty members and students, from both public and private academic institutions in Oman, who participated in this survey.

# data[].attributes.attachment_link.title
The Quality of Selected Online Learning Platforms and Their Effect on Education in the Sultanate of Oman

# data[].attributes.attachment_link.description
The COVID-19 pandemic has led to the largest interruption in education around the world, affecting some 1.6 billion students. Classroom education stopped, to be replaced by online e-learning platforms. In higher education, e-learning is made available through recorded lectures, with online platforms becoming a significant part of the overall system.

# data[].attributes.attachment_link.readable_text
Previous Article in Journal

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited.

Feature Papers represent the most advanced research with significant potential for high impact in the field. Feature Papers are submitted upon individual invitation or recommendation by the scientific editors and undergo peer review prior to publication.

The Feature Paper can be either an original research article, a substantial novel research study that often involves several techniques or approaches, or a comprehensive review paper with concise and precise updates on the latest progress in the field that systematically reviews the most exciting advances in scientific literature. This type of paper provides an outlook on future directions of research or possible applications.

Editor's Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to authors, or important in this field. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.

Department of Electrical and Computer Engineering, Ajman University, 346 Ajman, UAE

Department of Computer Engineering, Umm Al-Qura University, Mecca 715, Saudi Arabia

Department of Electrical Engineering, Information Technology University, Lahore 54700, Pakistan

James Watt School of Engineering, University of Glasgow, Glasgow G12 8QQ, UK

Author to whom correspondence should be addressed.

Received: 30 November 2020 / Revised: 8 January 2021 / Accepted: 18 January 2021 / Published: 26 January 2021

With the world becoming flat with fluid boundaries, engineers have to be global in their outlook and their pedigree. Due to the need for international acceptance of engineering qualification, the incorporation of Outcome-Based Education (OBE) has become common and global accreditation treaties such as the Washington Accord have been ratified. Further, it becomes important, especially for an engineering university with a global outlook preparing its students for global markets, to ensure that its graduates attain the planned outcomes. Additionally, the higher education institutions need to make sure that all the stakeholders, including students, parents, employers, and community at large, are getting a quality educational service, where quality is categorized as (1) product-based ensuring that the graduate attained the planned outcomes and skills, and (2) process-based keeping an eye on whether the process is simple, integrated, and efficient. The development of quality movements, such as Total Quality Movement (TQM), Six Sigma, etc., along with quality standards such as ISO 9001 has been instrumental in improving the quality and efficiency in the fields of management and services. Critical to the successful deployment of a quality culture is the institutionalization of an integrated Quality Management System (QMS) in which formally documented processes work according to the Vision and Mission of an institute. At the same time, commitment to Continuous Quality Improvement (CQI) to close the loop through effective feedback, would ensure that the planned outcomes are attained to the satisfaction of all the stakeholders, and that the process overall is improving consistently and continuously. The successful adoption of quality culture requires buy-in from all the stakeholders (and in particular, the senior leadership) and a rigorous training program. In this paper, we provide a review of how a QMS may work for the provision of quality higher education in a 21st-century university. View Full-Text

▼ Show Figures

Figure 1

Naveed Bin Rais, R.; Rashid, M.; Zakria, M.; Hussain, S.; Qadir, J.; Imran, M.A. Employing Industrial Quality Management Systems for Quality Assurance in Outcome-Based Engineering Education: A Review. Educ. Sci. 2021, 11, 45. https://doi.org/10.3390/educsci11020045

AMA Style

Naveed Bin Rais R, Rashid M, Zakria M, Hussain S, Qadir J, Imran MA. Employing Industrial Quality Management Systems for Quality Assurance in Outcome-Based Engineering Education: A Review. Education Sciences. 2021; 11(2):45. https://doi.org/10.3390/educsci11020045

Chicago/Turabian Style

Naveed Bin Rais, Rao, Muhammad Rashid, Muaaz Zakria, Sajjad Hussain, Junaid Qadir, and Muhammad A. Imran. 2021. "Employing Industrial Quality Management Systems for Quality Assurance in Outcome-Based Engineering Education: A Review" Education Sciences 11, no. 2: 45. https://doi.org/10.3390/educsci11020045

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

Back to Top Top

# data[].attributes.attachment_link.title
Employing Industrial Quality Management Systems for Quality Assurance in Outcome-Based Engineering Education: A Review

# data[].attributes.attachment_link.description
With the world becoming flat with fluid boundaries, engineers have to be global in their outlook and their pedigree. Due to the need for international acceptance of engineering qualification, the incorporation of Outcome-Based Education (OBE) has become common and global accreditation treaties such as the Washington Accord have been ratified.

# data[].attributes.attachment_link.readable_text
With a lack of consensus of what evaluation is within the field of evaluation, there is a difficulty in communicating to nonevaluators what evaluation is and how evaluation differs from research. To understand how evaluation is defined, both evaluators and researchers were asked how they defined evaluation and, if at all, differentiated evaluation from research. Overall, evaluators believed research and evaluation intersect, whereas researchers believed evaluation is a subcomponent of research. Furthermore, evaluators perceived greater differences between evaluation and research than researchers did, particularly in characteristics relevant at the beginning (e.g., purpose, questions, audience) and end (e.g., rendering value judgments, disseminating results) of studies. This study suggests that greater consensus on a definition of evaluation is needed to be able to distinguish the field and discipline of evaluation from related fields and to be able to communicate this information to nonevaluators.

References

Alkin, M. C., Christie, C. A. (2005). Unraveling theorists' evaluation reality. New Directions for Evaluation, 2005(106), 111-128.
Google Scholar

Castro, M. P., Fragapane, S., Rinaldi, F. M. (2016). Professionalization and evaluation: A European analysis in the digital era. Evaluation, 22(4), 489-507.
Google Scholar

Center for Open Science . (n.d.). Retrieved from https://cos.io/prereg/
Google Scholar

Hackbarth, D., Gall, G. B. (2005). Evaluation of school-based health center programs and services: The whys and hows of demonstrating program effectiveness. The Nursing Clinics of North America, 40(4), 711-724, x.
Google Scholar

Mason, S., Hunt, A. (2018). So what do you do? Exploring evaluator descriptions of their work. American Journal of Evaluation, 109821401876704. https://doi.org/10.1177/1098214018767049
Google Scholar

Poth, C., Lamarche, M. K., Yapp, A., Sulla, E., Chisamore, C. (2014). Towards a definition of evaluation within the Canadian context: Who knew this would be so difficult? Canadian Journal of Program Evaluation, 29(1), 87-103.
Google Scholar

Preskill, H., Russ-Eft, D. F. (2005). Building evaluation capacity: 72 activities for teaching and training. Sage.
Google Scholar

Rallis, S. F. (2014). When and how qualitative methods provide credible and actionable evidence: Reasoning with rigor, probity, and transparency. In Christie, C. A., Mark, M. M., Donaldson, S. I. (Eds.), Credible and actionable evidence: The foundation for rigorous and influential evaluations (2nd ed., pp. 137-156). Sage. https://doi.org/10.4135/9781483385839
Google Scholar

Yarbrough, D. B., Shulha, L. M., Hopson, R. K., Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage.
Google Scholar

# data[].attributes.attachment_link.title
What Is Evaluation?: Perspectives of How Evaluation Differs (or Not) From Research - Dana Linnell Wanzer, 2021

# data[].attributes.attachment_link.description
With a lack of consensus of what evaluation is within the field of evaluation, there is a difficulty in communicating to nonevaluators what evaluation is and how evaluation differs from research. To understand how evaluation is defined, both evaluators and researchers were asked how they defined evaluation and, if at all, differentiated evaluation from research.

# data[].attributes.attachment_link.readable_text
1. Create an ecommerce website.

Use our Website Builder to generate a mobile-responsive store for your industry with seamless site navigation, secure checkout, and more.

2. Help people find you.

Constant Contact helps you spread the word through email, social media, SEO and other forms of online marketing⁠-all from one place.

3. Use email to boost loyalty.

With email marketing tools like ours, you can segment subscribers based on different criteria and send them content they'll want to open.

4. Track your performance.

Constant Contact's analytic tool gives you detailed visitor metrics in real time⁠-allowing you to quickly identify and pounce on trends.
Get started

# data[].attributes.attachment_link.title
Email Marketing Software | Constant Contact

# data[].attributes.attachment_link.description
Use our Website Builder to generate a mobile-responsive store for your industry with seamless site navigation, secure checkout, and more. Constant Contact helps you spread the word through email, social media, SEO and other forms of online marketing⁠-all from one place.

# data[].attributes.attachment_link.readable_text
To understand the concept of Program Evaluation, it is essential to know its difference with Research, as most often these two concepts are confused between each other. A common question that is raised - is research and evaluation same or different? Unfortunately, there is a no easy answer for this. There has been alternative theorisation among experts in establishing the distinction between research and evaluation. There are various schools of thought or typologies that exist which view research and evaluation from different lenses. The four different typologies are as follows -
Evaluation as a sub-section of research
This school of thought is premised on the notion that - Doing research does not necessarily require doing evaluation. However, doing evaluation always requires doing research.
Research as a sub-section of Evaluation
The second school of thought on the distinction between research and evaluation is that research is only a sub-section of evaluation. According to this notion, the research part of evaluation involves only collecting and analysing empirical data.
Research and Evaluation not mutually exclusive
The third school of thought views research and evaluation as two unrelated variables that are not mutually exclusive. An activity can be both research and evaluation - or neither. Research is about being empirical. Evaluation is about drawing evaluative conclusions about quality, merit or worth.
Research and evaluation are dichotomous
Another school of thought considers research and evaluation as two completely separate stream of producing knowledge. Evaluation is viewed as more interested in specific, applied knowledge, and more controlled by those funding or commissioning the evaluation. Research on the other hand, is considered as interested in producing generalisable knowledge which are theoretical and controlled by the researchers.

# data[].attributes.attachment_link.title
What is the difference between Evaluation and Research?

# data[].attributes.attachment_link.description
To understand the concept of Program Evaluation, it is essential to know its difference with Research, as most often these two concepts are confused between each other. A common question that is raised - is research and evaluation same or different? Unfortunately, there is a no easy answer for this.

# data[].attributes.attachment_link.title
CRADLE Seminar Series: Joanna Tai - 26 February 2019 - DeakinAir

# data[].attributes.attachment_link.authors[].name
Arbor Educational & Clinical Consulting Inc.

# data[].attributes.attachment_link.title
What is program evaluation?: A Brief Introduction

# data[].attributes.headline
CDC-Elearning essentials: A guide for creating quality elearning

# data[].attributes.subject
CDC-Elearning essentials: A guide for creating quality elearning

# data[].attributes.headline
National Standards for Quality Online Courses

# data[].attributes.subject
National Standards for Quality Online Courses

# data[].attributes.attachment_link.provider_name
National Standards for Quality Online Learning

# data[].attributes.attachment_link.title
Quality Online Courses - National Standards for Quality Online Learning

# data[].attributes.attachment_link.description
The National Standards for Quality Online Programs provide a framework for K-12 schools, districts, state agencies, statewide online programs and other interested educational organizations to improve online and blended learning programs.

# data[].attributes.headline
Rapid course migration: Comparing the instructors' self-efficacy

# data[].attributes.subject
Rapid course migration: Comparing the instructors' self-efficacy

# data[].attributes.headline
What is effective online teaching and learning

# data[].attributes.subject
What is effective online teaching and learning

# data[].attributes.headline
Implementing QA system for open education..

# data[].attributes.subject
Implementing QA system for open education..

# data[].attributes.attachment_link.readable_text
Copyright

License

Published in Asian Association of Open Universities Journal. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode

Introduction

Quality has been a major importance for higher education ( Inglis, 2005), and the provision of quality higher education is a major challenge to address by open universities (OUs) and open and distance learning (ODL) institutions. This study attempts to better understand the implementation of quality assurance (QA) system in three Asian OUs, namely University of the Philippines Open University (UPOU), Philippines; Universitas Terbuka (UT), Indonesia and Allama Iqbal Open University (AIOU), Pakistan. Quality has existed in these three OUs since their beginnings of operations, and QA system has been further implemented. As new QA methods emerge and ODL systems evolve, OUs also adapt to new ways of assuring quality to meet the expectations of students and the stakeholders. This research aims to reveal QA implementation in the three OUs in terms practices, challenges and rooms for improvement and learning lessons that may be relevant to the OUs and relevant ODL systems. Findings of this study are to present insights into exploring new ways of effectively implementing QA system in ODL. This study was conducted prior to the COVID-19 outbreak, and thus a section on the implications of COVID-19 pandemic on ODL directions has been added.

Meaning of quality

The meaning of quality has been much debated, and stakeholders, educational institutions and students may hold different conceptions of quality. "Quality is in the eye of the beholder" ( Harvey and Green, 1993; Brockerhoff et al., 2015), and it will remain an important challenge for higher education. Quality is a relative concept as "it means different things to different people" ( Harvey and Green, 1993). Harvey and Green's conceptualization of quality in higher education has been widely used in which quality is categorized into five discrete but interrelated ways of viewing quality as exceptional, as perfection (or consistency), as fitness for purpose, as value for money and as transformative ( Harvey and Green, 1993). The true meaning of quality in higher education is not as simple as it has been generally perceived. Van Kemenade et al. (2008) define quality in terms of value systems and suggest four value systems on quality and quality management for analysis, namely: process control, continuous improvement, commitment and breakthrough. Within these value systems, process control relates to compliance to rules, procedures and standards and is relevant to the industrial management of distance education, and continuous improvement has been added to have no limit to the level of improvement to achieve ( Van Kemenade et al., 2008).

Difficulties in defining the meaning of quality pose challenges in development of methods of assessing the quality, in which the quality assessment methodology need to meet the criteria for public accountability, academic credibility, practical feasibility and quality audit ( Harvey et al., 1993). It may be unnecessary to have a universal definition of quality, as quality and QA may be culture context ( Schindler et al., 2015), and that there seems to be no agreement on universal definition, model, and practices of QA in higher education ( Ryan, 2015). Each institution may have their own definitions, models and practices that suit to their respective cultures and stakeholders' requirements. OUs and ODL institutions establish their own QA processes, follow accreditation systems and implement internal QA systems that suit the culture contexts and meet stakeholders' expectations of the respective countries.

Quality teaching and learning or the pedagogical techniques to produce learning outcomes for students is one important educational goal to achieve in higher education. Achieving quality teaching and learning is a complex endeavor involving multiple dimensions, including design of curriculum and course content, learning contexts, use of feedback, assessment of learning outcomes, learning environments and student support services ( Hénard and Roseveare, 2012). In an ODL system, a comprehensive and an integrated online student support system is mandatory to maintain the quality of open and distance learning ( Mir, 2017; Mir et al., 2019). ODL requires more endeavors in terms of logistics, supply chain management and supporting activities, such as laboratories to support effective teaching and learning of science online at a distance ( Kennepohl and Shaw, 2010). Quality teaching and learning a variety of subjects in ODL institutions may be difficult to define and standardize in terms of process, as different subjects may require different approaches to teaching and learning, pedagogical foundations and designs depending on students' needs, delivery and logistic requirements. Quality has been defined and understood from many different angles, that may have consequences on how QA policies and operational systems are to be implemented. Elken and Stensaker (2018) have conceived the idea of "quality work" in higher education and suggested multiple approaches to quality in higher education embracing the dimensions of management, work and quality.

Quality assurance in open and distance learning

QA in ODL has been a field of practice that has dominated educators and stakeholders during the past several decades. The meaning of quality has ramifications on the practice of QA in ODL, including massive open and online curses (MOOCs). The transformation of ODL into online learning has taken place as part of developments in the use of new technology in teaching and learning. Technology has enabled the emergence of online learning in higher education during the last decade of the past century ( Anderson and Elloumi, 2004). Online learning continues to flourish with the introduction and widespread use of MOOCs by campus-based higher education institutions to reach wider audiences ( Baturay, 2015).

Effort on QA in online learning has been strengthened through publication of guidelines ( Uvalić-Trumbić and Daniel, 2013; Grifoll et al., 2010; Huertas et al., 2018). Research on QA for online higher education programs has been conducted identifying the system of indicators for the quality assessment for online higher education programs ( Marciniak, 2018) and using PDPP (planning, development, process and product) evaluation model ( Zhang and Cheng, 2012). Effective implementation of QA system in OUs, ODL and online learning system has been daunting tasks and shared responsibilities by the management, people, and stakeholders involved in ODL( Alfonso, 2015; Belawati and Zuhairi, 2007; Bibi et al., 2018; Jamandre, 2011; Jung et al., 2013).

Research findings on the development of QA in Asian ODL has revealed variations on the level of QA policy integration in the overall national QA in higher education policy, and these pose challenges for higher education institutions and governments to ensure quality graduates for the job market ( Jung et al., 2011). Jung (2012) has further analyzed Asian learners' perception of ODL in terms of three domains, namely supportive domain (faculty supports, student supports, information and publicity), pedagogical domain (teaching and learning, course development, teaching and learning, interactive tasks, evaluation and assessment) and environmental domain (infrastructure, internal QA mechanism and institutional credibility). Benchmarking and assuring the quality of graduates for employability becomes priority of many higher education institutions and governments. QA concepts and implementations in ODL system have also evolved to respond to changing students' needs, technological developments and transformation into digital and online learning ( Arinto, 2016; Vlachopoulos, 2016). It is worth noting the relevance of variations in QA policy integration, transformation into online learning and methods to address these issues in the contexts of Philippines, Indonesia and Pakistan.

Quality is everyone's concern, and QA movement has encouraged educational organizations and governments to develop guidelines for use by higher education institutions. Organizations with stakes in ODL such as UNESCO, COL and AAOU have produced toolkits and guidelines to help members achieve their QA implementation ( COL, 2009; AAOU, 2019). Stakeholders with interests in assuring QA of online learning also develop guidelines to quality in online learning for reference by interested parties ( Uvalić-Trumbićand Daniel, 2013). Despite the effort, the integration of quality management system, accreditation system, internal assessment and external quality review has remained a challenge for many ODL institutions. There are specific matters to address in QA for ODL because of the specific characteristics of the ODL system ( Stella and Gnanam, 2004).

QA in higher education can be analyzed at three different levels: individual, organizational and system levels ( Brockerhoff et al., 2015). Analyses at individual and organizational levels can be more readily conducted, as it is generally under the supervision and monitoring of the leaders of the institution. Analyses at system level go further beyond individuals and the organization, may include outcomes that are difficult to assess and can be more complicated to evaluate, as it needs to involve external governance arrangements and efficiencies. Governments also refer to published QA frameworks as reference to establish their national QA systems for higher education.

Open universities have been under pressures to meet the students' and stakeholders' expectations on quality. A number of studies on QA in online learning and teaching have been conducted to address the issue of quality along with the changing higher education landscape as the results of the use of online technologies in teaching and learning ( Kooi and Ping, 2012; Sembiring and Rahayu, 2019). A study in Malaysia has indicated the use of quality tools to improve student retention supporting process ( Chuah and Lim, 2018). QA agencies have played increasingly important roles to ensure quality and public accountability of higher education programs ( Zhang and Patil, 2017: Stensaker, 2018). Perceptions of quality on ODL varies from different stakeholders and students across different regions ( Jung, 2012), and so diverse approaches to QA in ODL across Asia has been implemented ( Jung et al., 2011). Various ODL systems have intensified their effort to integrate technology enhanced learning into their existing systems ( Volungeviciene et al., 2014) and carefully addressed the issues of the convergence of open learning philosophies, distance education pedagogies and e-learning technologies ( Arinto, 2016; Vlachopoulos, 2016).

Quality assessment and accreditation in the three Asian open universities

Quality assessment and accreditation in the three Asian OUs in Philippines, Indonesia and Pakistan have been implemented, and they share commonalities and differences due to diverse society needs and stakeholders' expectations. For Southeast Asia, at the regional ASEAN (Association of South East Asia Nations) level in which the Philippines and Indonesia are member countries, assuring the quality of higher education has become one of the common goals of governments through joint collaborative efforts. ASEAN-wide guiding principles on QA and recognition of competency certification system have been published to provide reference for implementation by ASEAN member states ( ASEAN Secretariat, 2016). The ASEAN University Network has published a guideline for use by the member as well as other interested institutions ( AUN, 2016). The 2006 ASEAN QA framework provides the basis for developing understanding for the recognition of competency certification systems, establishing guiding principles and protocols for developing processes for recognition of these systems and reviewing the state and development needs for QA among ASEAN member countries ( ASEAN Secretariat, 2016).

QA in higher education shares common principles in different countries ( Al Tobi and Duque, 2015; Dotong and Laguador, 2015). For higher education in Philippines and Indonesia, QA in ODL may be viewed as continuous improvement process ( Ruiz and Junio-Sabio, 2012; Tadjudin, 2001). In Indonesia, the Ministry of Education and Culture has continuously updated guideline for internal QA System for implementation by higher education institutions ( Kemenistekdikti, 2018). The Philippines has had a long history of QA system and accreditation for higher education, starting in 1957 with the establishment of the country's first accrediting agency, the Philippine Accrediting Association of Schools, Colleges and Universities ( Corpus, 2003). The Philippines government has continuously improved its QA system and has been moving towards the development of self-regulation policies, mechanisms and procedures ( Ruiz and Junio-Sabio, 2012).

Indonesia has had a long history, large number of students, different types of public and private higher education institutions and a diverse system of higher education in terms of governance, access and quality ( Logli, 2016). The national higher education system of Indonesia enrolls over 6.9m students, and the system employs more than 275,000 lecturers and professors ( Ahmad, 2018). These diverse groups and characteristics of higher education institutions in Indonesia pose specific challenges in terms of diversity in quality and ways to address them to ensure the Indonesian higher education system meets quality criteria and public expectations. The formal QA movement in Indonesian higher education has begun recently during the beginning of the 2000s with the publication of Guidelines for Internal QA System for implementation by public and private higher education institutions in Indonesia ( Kemenristekdikti, 2018). QA system necessitates internal and external assessment, and higher education institutions need to ensure that quality criteria are met. An internal QA unit in each institution has been established to develop internal guidelines, standards and procedures for implementation within the institution, and mechanisms for audit are developed. Externally, higher education institutions need to assure that the products and services meet quality criteria set by its stakeholders. External assessment conducted by certification and accrediting agencies or external quality review agencies is important to ensure that the internal quality criteria meet certain level of benchmark to external quality criteria.

The Government of Pakistan through its Higher Education Commission has strong commitment to QA and enhancement of higher education through the publication of Quality Assurance Manual for Higher Education in Pakistan as reference for higher education system to implement QA system. Earlier, the Pakistan Quality Assurance Agency (QAA) was established in 2004 within the Commission of Higher Education "as a specialized body to introduce and encourage the development of quality culture in higher education". The Pakistan higher education system follows the guidelines and manuals of QA issued by the Pakistan Higher Education Commission ( Batool and Qureshi, 2008). Quality higher education is a top priority for the Government of Pakistan to ensure quality education for the citizens ( Usmani et al., 2012) through which the QAA established Quality Enhancement Cells within the higher education institutions to conduct self-assessment reports of academic programs ( Usmani and Khatoon, 2016).

Method of the study

This study employed qualitative method involving analyses of documents, focus group discussion and interviews with management and staff. The documents for analysis include institutional documents, and other publications of UPOU, UT and AIOU. Focus group discussions and interviews with management and staff were conducted to have in-depth understanding and clarify matters. The authors spent one month at UPOU analyzing the documents, collecting data and having discussions and interviews with management and staff. Further data collection activities were also conducted in UT and AIOU to ensure that recent information could be updated and latest development of the three institutions under study could be added.

This research aims to understand how QA systems are implemented in UPOU, UT and AIOU. The study focused on revealing practices, challenges and rooms for improvement in terms of QA implementation in the three Asian OUs. This study replicated the research methods of two previous comparative studies of various aspects of OUs in Sri Lanka, Taiwan and Indonesia, and thus some of the findings on particular aspects of this present study on UT, particularly on the characters and good practices of the institution, were adapted and updated from these previous two studies on supporting learners in ODL ( Zuhairi et al., 2019) and on empowering lifelong learning ( Zuhairi et al., 2020).

During this present study in UPOU in November 2019, the authors conducted interviews and focus group discussions on QA with senior administrators, managers, academic members of staff and QA support staff. Furthermore, individual authors also conducted interviews with management and staff of the respective institutions to get updates of recent developments of the three OUs under investigation. List of people consulted at UPOU through focus group discussion and interviews could be found in Appendix (See Appendix. List of people consulted at UPOU Philippines).

Findings and discussion

Findings of this study provided information about QA implementation and the potential directions of OPOU, UT and UT as modern OUs in providing quality ODL programs to students. Comparing OUs has been a difficult endeavor, as each institution has been designed distinctively by its stakeholders to address the specific needs of the respective societies. Yet there are common indicators that the three Asian OUs may be compared to provide insights on how they function as effective OUs accessible by large number of students in flexible manner.

The unique differences of the three Asian OUs were observed. Although each open university has been designed to serve their respective national clients, ODL system has also allowed the citizens of the respective countries living globally to have access to their ODL programs online. The three OUs have been made capable of accommodating everybody into the program, including those noncitizens. UPOU is predominantly intended to provide post-graduate programs at the master and doctoral levels, with fewer undergraduate programs. UT is a comprehensive open university offering diploma and degree at undergraduate and post-graduate levels, with most of the students studying for undergraduate degrees. As a comprehensive open university, AIOU offers undergraduate and post-graduate programs and even accommodates large number of students taking high-school certificates. Table 1 illustrates indicators of the three OUs in terms of date established, area served, language of instruction, client, faculty, programs offered, head office, regional offices and learning centers and QA system.

Good practices of the three Asian open universities

The three Asian OUs share common practices in serving their respective societies appropriately. As ODL pioneers in their respective countries, they transform themselves adapting to the new landscape of online digital learning. UPOU has jumped right from the beginning into becoming an open online university. UT and AIOU have diverse groups of students with different access to technology and thus accommodate both traditional distance students and online students. The three OUs are state institutions addressing the common challenge for quality flexible learning for their respective students and leading their roles as open, distance and online learning nationally. Table 2 indicates good practices of the three Asian OUs in terms of institution character and responsibility, core value, programs offered, missions, funding and tuition fees, and national roles.

Challenges of the three Asian Open Universities

The three OUs share common challenges, particularly pressures to accommodate even larger number of ODL students. Integrating online learning and the use of new technology offer greater flexibility for OUs, but there are constraints for those students that have limited access to new technology and online services. The use of new technology confronts OUs to train staff and students and familiarize them with the technology to facilitate learning. There are also the issues of openness and lifelong learning. How open an OU depends very much on the government policies and the institution capabilities to serve various kinds of students. Integrating learning and teaching online is a further challenge because of the different groups of students the OUs serve and the academic staff readiness with the new ways of teaching and learning online. Table 3 illustrates the challenges of three Asian OUs in terms of student numbers, integrating online learning, access to services, new technology, staff training, academic staff workload, admission policy and duration of study.

Rooms for improvement for the three Asian open universities

There are rooms for improvement relevant to individual institutions under study. Each of the OUs have specific strategies in providing student support due to the differences in student characteristics. Stakeholders generally have high expectations in the capability of OUs to accommodate large number students in higher education. Table 4 indicates rooms for improvement for three Asian OUs in key areas of student support, participation in higher education, lifelong learning and partnerships.

Good practices in QA system in the three Asian open universities

OUs have the natural characters of openness, flexibility and capacity to reach wider audience through open, distance and online learning. QA has been an important instrument to improve the quality of ODL with large number of students. An earlier study on QA in Asian ODL system has revealed variability in terms of the level of QA policy integration in the overall national QA in higher education policy framework as well as QA purpose, policy frameworks, methods and instruments, despite obvious commonalities that underpin their QA efforts ( Jung et al., 2011). The QA movement in higher education beginning in the 1990s has had constructive effects on OUs, including those in UPOU, UT and AIOU. It was during those times that many OUs started to formally address the challenge for quality and establish QA system for implementation to meet stakeholders' expectations.

UPOU was established in 1995 with the purpose to democratize access to quality higher education through ODL, being one of seven universities and an autonomous college of University of the Philippines (UP) system offering mostly post-graduate and a few undergraduate programs in social sciences and applied disciplines ( Arinto, 2016). UPOU is one of the pioneers in open and distance e-learning as an innovative approach for delivering effective ODL to anyone in flexible manner using the digital technologies along with other forms of learning materials and support system ( Oruga and Bagos, 2018). QA mechanism has existed in UPOU right from the beginning, and QA implementation has recently been strengthened involving external assessment internationally.

For UT, as the only state open university established in 1984, it has since consistently followed the rules, guidelines and procedures set by its parent Ministries of Education and Culture and of Finance to offer study programs to its ODL students. A systematic approach to QA in UT formally began in 2001 with the establishment of a QA Committee, and then followed in 2003 by the establishment of a QA Center, a unit dedicated to managing the implementation of a comprehensive QA system. UT began to adopt the Asian Association of Open Universities (AAOU) QA Framework 2001 as quality policies for implementation. The UT core activities, products and services were defined in terms of quality standards to be met. Standardized procedures were documented for implementation in relevant units by managers and persons in charge. QA is a continuous improvement process, and it necessitates transformation of the people within the institution in terms of quality culture and commitment to QA ( Belawati and Zuhairi, 2007). The UT has been challenged to transform itself and adapt to the new landscape of ODL, adopting online approach to teaching and learning in line with advances in the use of information and communication technology ( Darojat, 2018).

UT is a pioneering ODL institution, serving ODL students using blended online learning. Many students opt for the face-to-face tutorial support provided by the UT, and there has been increasingly more students prefer online tutorial support that provides greater flexibility on the parts of the students. Implementing QA system in UT is a formidable task because of the different groups of students to serve and the geographical coverage of the students. Systematic approach to QA has since been implemented involving both internal and external assessment. The ISO 9001 certification has been implemented since 2005 for the quality management system of its industrial management processes, and the ISO 27001 for information technology security management has recently been attempted for implementation and certification. Accreditation of study programs and the institution have been conducted by the National Accreditation Board for Higher Education (BAN-PT). The International Council for Open and Distance Education (ICDE) has further conducted quality reviews every 3-5 years since 2005 to ensure that the UT practice of ODL meets international standards.

Established in 1974, AIOU has been the pioneer in nonformal and ODL in Pakistan, offering programs at both undergraduate and post-graduate levels as well as secondary and high school certificates for adult learners ( Iqbal and Ahmad, 2010). QA is an important concern of the institution to ensure the provision of quality education for the people of Pakistan. AIOU has important roles in empowering lifelong learning among Pakistani citizens and ensuring access to quality higher education among high school graduates and adult learners.

The three Asian OUs share common good practices in QA system, although each OU may have referred to different framework and address to the respective need of the stakeholders. QA has been implemented with great care and serious considerations, involving both internal and external quality reviews. Table 5 illustrates good practices in QA system implemented in the three Asian OUs covering their internal mechanism, QA framework and mechanism, internal and external assessment, accreditation and international quality reviews.

Challenges in QA system in the three Asian open universities

The three OU has taken quality assessment seriously. For UPOU, external quality reviews have been introduced recently. External assessment nationally is implemented to ensure that UPOU programs and courses meet stakeholders' expectations, while external assessment internationally has been conducted through quality reviews by ASEAN University Network (AUN). For UT, QA is a delicate process with the internal quality audit functioning to establish awareness, good attitudes and confidence in QA and to ensure that people work in compliance to guidelines and procedures. As a systematic continuous improvement effort, QA aims to develop quality work culture among people within the institution. External quality assessment aims to ensure public confidence of the QA system. At AIOU, the QA Framework developed by the Commonwealth of Learning (COL) has been adopted with external assessment conducted by the Higher Education Commission (HEC). External academic reviews at national level have been conducted at program level by HEC. For AIOU, QA has involved the internal mechanism and the external benchmark and assessment by the Higher Education Commission.

The three Asian OUs have consistently implemented their respective QA system and addressed their challenges in different stages of QA development. Both UPOU and AIOU share common challenges in QA system, while UT has started QA earlier needs to strengthen its QA system to add values to the institution. Table 6 illustrates the challenges in QA implementation in terms of system, shared responsibilities, values, work culture, compliance and internal and external assessment. Quality assessment is an important component of any QA system, as it informs the institution and stakeholders the quality level achieved referring to certain criteria and benchmark. Internal assessment is important as feedback for continuous improvement and as a tool to appreciate effort on QA by the institution to its dedicated people implementing the system. External assessment using commonly accepted benchmark and external quality criteria is important to ensure public confidence. Table 6 illustrates challenges in QA system in terms of shared responsibilities, quality values and work culture, compliance and internal and external assessment.

Rooms for improvement in QA system for the three Asian open universities

The three Asian open universities share common rooms for improvement in QA system, despite the fact each of OUs has their own historical origins and paths for development. There are two major areas for improvement that the three Asian OUs share, namely shifting from distance to online learning QA system and from audit to analytics-based QA system. These findings confirm the importance of QA as priorities of the three Asian OUs. Table 7 presents rooms for improvement in QA system in terms of shifts from distance to online learning and from audit-based to analytics-based QA system.

Implications of the post COVID-19 pandemic on QA implementation in ODL

The study was completed just prior to the outbreak of theCOVID-19 pandemic, that have since brought about significant consequences on higher education and ODL systems worldwide, including the three open universities under study. The COVID-19 pandemic has grown quickly to affect global operations of higher education and has called for the need for the deployment of emergency e-learning and the consequences for the securitization of higher education ( Murphy, 2020). Stakeholders and organizations have been quick to address the issue of education affected by the COVID-19 pandemic through publications of policy ideas, guidelines, resource sharing initiatives, sharing of practices and action plans to respond to the on-going crisis ( UNESCO, 2020a, b; World Bank, 2020, 2020a; OECD, 2020; IAU, 2020; Belawati and Nizam, 2020). These resources have helped higher education institutions to respond appropriately to the emergency necessities for e-learning implementation worldwide. These sudden transformations into online learning may have serious consequences on the quality assurance of higher education and ODL system that needs to be addressed carefully.

As earlier stated by Volungeviciene et al. (2014), OUs world worldwide vary in terms of the level technology-enhanced learning integration into the organization. The post COVID-19 experience may have different implications on OUs in different ways. The University of the Philippines (UP) system is a large multi-campus university system, first established in 1908, with seven campus-based universities in Diliman, Los Baños, Manila, Visayas, Mindanao, Baguio, Cebu and satellite campuses across the country and the UPOU headquartered in Los Baños ( University of the Philippines, 2020). UPOU is already an open online university, and thus all students already use online learning support and services. UPOU should be able share good practices in online learning with the campus-based UP system. The COVID-19 pandemic has significant implications on higher education in the Philippines, including the UP system, pushing the campus-based system into applying remote learning, modular courses, MOOCs, televised courses and new technology ( Llaneta, 2019). The UPOU experience in online learning should benefit the UP system in addressing its post-COVID19 higher education practice.

During the pre-COVID-19 pandemic, both UT and AIOU have gone through the continuous progress of transformation into online learning, with constantly increasing number of students using their online learning services. For UT, less than one-half of its student population uses online learning support, and for AIOU the proportion of students using online learning support has even been smaller. During COVID-19, AIOU introduced the concept of end term assessment (ETA), a kind of open-book exam to be performed at homes, as an alternate of final examination which was being held physically in examination halls before COVID-19. The workshop component of all teacher training and post-graduate programs of more than 100,000 students is being conducted online through virtual classroom environment and the AAGHI Learning Management System (LMS) in synchronous mode. All these systems are being managed and hosted by AIOU. The COVID-19 experience may highly likely drive both OUs and relevant higher education institutions to accelerate their online learning transformation processes that benefit the institutions and their students.

The COVID-19 pandemic has prompted UT and AIOU to take actions adapting many of its academic services such as learning support and examination, as well as administrative services and quality audit activities, shifting from the face-to-face to mediated or web-based activities. Take UT as an example, during the COVID-19 pandemic the face-to-face tutorial services have been replaced with the web-based tutorials. Similarly, the paper-based semester examinations have been replaced with assignments or online examinations. These policy changes were to be communicated speedily with all students and staff. These changes in modes of services had to be understood precisely by the management and all staff in regional offices, as well as students and tutors in various regions. New orientations on these changes in the teaching, examination and assignment requirements were to be conveyed to the students and tutors timely to ensure common understanding among themselves.

The COVID-19 pandemic has had significant consequences on face-to-face higher education institutions and OUs. Campus-based universities need to adjust to the new normal for adopting e-learning methods to replace their traditional classroom teaching and learning. For Indonesia, higher education institutions have undergone sweeping digital transformation through adoption of online learning and new technology to replace the traditional face-to-face mode of learning in the light of the COVID-19 pandemic, and just within less than a month 98% of 4,621 Indonesian higher education institutions have suddenly gone online in response to the COVID-19 disruption ( Belawati and Nizam, 2020; Nizam, 2020). These phenomena of campus-based universities transforming into online learning are also taking place in Philippines, Pakistan and virtually in every country worldwide in reaction to the COVID-19 ( UNESCO, 2020a, b). Many campus-based universities have been involved in offering MOOCs, and they may have developed their necessities to improve and expand their distance learning capabilities along with its MOOCs. These post-COVID-19 technological transformations need to take various consideration perspectives related to digital literacy, pedagogy, digital learning materials, learning interactions, technology and assessment of learning ( Belawati and Nizam, 2020).

The post-COVID-19 pandemic confronts specific challenges in the implementation of QA in ODL. Educational organizations and stakeholders have further published specific guidelines on facilitating flexible learning and ensuring the quality of its provision to address the challenge of education during COVID-19 pandemic ( Huang et al., 2020a, b). For many higher education institutions and OUs, the COVID-19 pandemic has taught tough lessons to modernize and move forward even faster to become more technologically enhanced open and online universities able to reach various groups of students. For governments, they need to work out on policies that enable improved access through the development of open online learning policies that benefit their citizens. The OU and ODL systems historically have been established by their founders based on careful considerations on quality, access, equality and flexibility to ensure public trust and credibility. These principles have been continuously nurtured to ensure sustainability and ability to adapt to disruptions and address future challenges.

Conclusion

Significant effort has been put forth for implementing QA at UPOU, UT and AIOU. Findings of this study revealed the three OUs in terms of achievement, lessons and future directions. These findings would be helpful in identifying and charting the potential future action plans for continuous improvement in the individual OUs and their respective QA systems. These findings may also be of interest as reference by other OUs and ODL systems.

The three Asian OUs have their own achievements worth respecting.QA implementation has been a dedicated effort in the three Asian OUs, with seriousness of purpose, careful supervision, audits and involving management and staff to invest significant resources for the effort. Commitment to QA implementation has been a great start with the good spirit for continuous improvement. Management and staff work together as a team to ensure that the institution's vision, missions and goals be achieved through shared responsibilities. The three Asian OUs have developed an internal mechanism for QA, involving internal assessment as well as external assessment by the agencies responsible for ensuring quality higher education in the respective countries. UT has gone further in external quality assessment through the ISO 9001, 21001 and ISO 27001 certifications, study program and institution accreditations by the BAN-PT and reviews by a panel of quality experts of the ICDE. The missions and goals of OUs are constantly aligned to satisfy the needs of students and stakeholders. One important common value shared by the three OUs is that QA has been acknowledged as a shared responsibility by all.

Some lessons may be learnt from the three Asian OUs. Effective implementation needs significant effort, resource commitments and internalization among management and staff to develop quality awareness and values for continuous improvement. Human resources need to be trained and retrained to comply guidelines and procedures to provide services at the highest quality level. Internal and external assessments need significant effort and time to prepare and different external assessment conducted by different bodies require specific preparation, as each of the external bodies may have different information requirements and criteria for quality assessment. External bodies come at different time with different emphases on the aspects of quality under assessment. These external assessment regimes may cause lots of pressures to management as staff, as they consume significant amount of time and energy for preparation and follow up of their findings. All management and staff need to be directed towards the achievement of the goals of OUs as quality ODL providers in higher education.

There are possible future directions worth considering by the three Asian OUs that may also be relevant to other OUs, particularly in the use of technology in ODL. OUs worldwide, including the three Asian OUs, have committed investment and resources to transform themselves into modern OUs, which provide services on digital platform. Information technology has become the backbone and nerve center to provide both administrative and academic services to students with respects to their learning needs and styles. OUs are challenged to address these changing landscapes in the ODL provision on massive scales and global scope. New technology has allowed higher education institutions of all kinds, campus-based and open universities, to offer blended and online learning courses of high quality and relevance to society needs. Significant changes in the way higher education institutions offer high-quality programs on-campus and online are challenging OUs to adapt to the changing landscapes of online learning environment.

The Asian ODL systems have gone through phenomenal expansion during the past five decades, and there has since been growing public demand for ODL quality and accountability ( Jung et al., 2011).There are common characters in implementing QA system in the three Asian OUs in which QA system is a viewed as shared responsibilities adopted by their leaders, respected by both management and staff, and implemented by all people. Such great values have been internalized through a long process of trials and continuous improvement. Common beliefs are established and continuously refined to ensure sustainability. This study of three Asian OUs comes to final conclusions on the characters of their QA systems as illustrated in Figure 1.

Quality ODL system focuses on effectively meeting the needs of ODL students and stakeholders. The COVID-19 pandemic has compelled ODL system to be able to ensure sustainability though continuous adaptation and innovation ( World Bank, 2020a). Quality ODL puts students and their stakeholders first at the center of its QA and improvement effort, along with adaptability to respond to disruption, such as COVID-19. ODL institutions work closely with stakeholders to identify their needs and then develop programs and courses that are relevant to their needs and those of potential employers. ODL institutions keep assessing themselves the quality level which they are in, design action plans on how to improve them and then systematically implement those plans to improve quality. ODL institutions also continuously ask their students on their levels of satisfactions, and then decide which areas of services need further improvement and which areas of their strengths that they continue to maintain. Conclusions are made and elaborated to better understand the characters of the QA systems of the three Asian OUs.

First, the value "quality begins with inner self" seems to be universal merits implemented in OUs, and it is clearly observable in the three OUs in the way the people work and accomplish their activities. It is the intrinsic motivation for quality improvement within every individuals and units that has empowered people within these three Asian OUs to move forward with QA implementation with strong commitment, inspired by their dedicated leaders. Shared responsibilities are the key words for an effective QA system. "Quality begins with the inner self" is the fundamental social capital for consistent QA implementation in these OUs.

Second, quality is a multidimensional concept, borrowing the term by Vlachopoulos (2016), as it embraces a broad range of products, services, deliveries and philosophies, and it attempts to meet the needs and expectations of students and different groups of stakeholders with differing interests. It means different things to different people ( Harvey and Green, 1993). It is then the challenge of the OU leaders to empower all management and staff to address those conflicting interests and processes and ensure continuous quality improvement and enhancement of the OU system. Implementing a QA system implies challenging tasks of addressing multiple dimensions, aspects and meanings of quality from different perspectives and interests.

Third, QA is a systematic continuous improvement effort. QA as a system involves well-defined sets of principles, procedures and mechanisms organized to achieve the common goals of the institution. Effective QA system is supported with resources and tools needed for effective implementation by the people within the institutions. There seems to be no limit to continuous improvement, as the expectations of ODL leaders, staff, students and stakeholders will continue to increase from time to time. ODL institutions will carefully look at benchmark and good practices of others, and then develop systematic effort for continuous improvement and quality enhancement.

Fourth, QA implies effective mechanism and assessment, ensuring that different processes are carefully executed and assessed. The mechanism will define which processes to be conducted by whom using which resources. Effective QA system involves continuous quality assessment both internally and externally. Internal assessment is the foundation for continuous improvement, while external assessment goes further to serve as the external benchmark to ensure public credibility and meet the universally accepted good practices of ODL.

Fifth, QA is an attempt at exceeding the expectations of students and stakeholders. Good QA system is focused on customer satisfaction, and thus students and stakeholders will be the parties that most benefit from the QA effort. The expectations of students and stakeholders are dynamically changing to respond to professional competency requirements, societal needs and technological changes in the world of work. OU systems will continue to innovate to best serve the needs of their students and align themselves to the needs of employers and stakeholders for high caliber OU graduates.

Sixth, QA embraces a process of ODL transformation into online digital system and adaptability to disruption and the new ODL landscape. ODL system begins with the print-based correspondence system, moving to multimedia broadcast-based system, then to online digital system. These technological changes have been adopted by OUs, many of which have historically been established using the print-based platform. Modern OUs more recently founded during the digital age have naturally jumped into the digital online system, such as UPOU. And there are other OUs in the world that have transformed them gradually, and some of them more quickly to become digital online OUs. Both UT and AIOU particularly have been challenged to maintain and improve the quality of services to different groups of students with conflicting information technology literacies, one group being more technology literate than the other. Both OUs have gone through continuous balancing interests in adopting digital online system to serve their younger groups of students who are more digital literate and at the same time maintaining older groups of students who have restrictions on the use of online digital system. It is further complicated by the fact that there are diversities in terms of ease of access to online digital system due to geographic and infrastructure constraints.

The three Asian OUs under investigation, like many other OUs are constantly challenged to meet the expectations of their students and stakeholders through continuous improvement and quality enhancement. These OUs seem to have to work closely with their respective governments, other stakeholders and students to identify their needs and respond to their needs appropriately. QA is a highly dynamic process, and QA system is one of the instruments which can be effectively used to continuously improve and enhance the quality of ODL provisions.

To respond to the COVID-19 pandemic, the three OUs need to take necessary actions to ensure that their ODL services remain well implemented following quality guidelines. Strategic plans and policy directions may have to be redrawn to accommodate such disruptions, and action plans need to carefully address the changing needs of the students and stakeholders. AIOU may have to speed up its plans to expand its online access and services to its students. For UT, its roadmap to be a leading cyber university may have to be revisited and materialized even sooner ( Darojat et al., 2018), as online learning may have become the necessities of students in higher education in post-COVID-19. These efforts seem to be shared responsibilities among institutions, governments and relevant stakeholders.

Higher education institutions may have to rethink on their post-pandemic pedagogy ( Murphy, 2020). The COVID-19 pandemic has challenged higher education system across the world to shift to online teaching and learning mode almost overnight, enforcing many reluctant institutions and educators to change their traditional pedagogical approach to online teaching and learning environment ( Dhawan, 2020). A further study following the COVID-19 pandemic indicates that universities worldwide are moving faster towards online learning, and there seems to be stronger needs for the use and integration of technology enhanced learning into the system along with other needs for resources, staff readiness and student accessibility to technology ( Ali, 2020). The post-COVID-19 necessitates higher education institutions to transform into technology-based teaching and learning that benefit learners, in which courses may be offered on fully online, hybrid or augmented immersive residential modes ( Govindarajan and Srivastava, 2020). The world has been moving online fast, calling for higher education in digital age to move online ( Zorn et al., 2018), and the post-COVID-19 experience may expediate this online digital transformation of higher education even faster.

Historically because of its departure from the face-to-face higher education system, the OU and ODL system has been established on careful quality considerations to ensure public trust and credibility. In terms of QA, the sudden disruption of COVID-19 poses specific challenges, as new policies, systems and procedures need to be adapted to meet the COVID-19 teaching and learning environment. The online component of ODL services may not be affected as much as the traditional ODL system requiring face-to-face services, such as the face-to-face learning support, paper-based examination and administrative services. Thus, online open university may continue its operations more smoothly than their traditional OU counterparts with face-to-to-face services. It remains to be seen how the landscape of higher education and online learning will continue to change for the long-term in the post-COVID-19 outbreak, and it is certain that QA remains at the forefront to ensure that the expectations of students and stakeholders can be exceeded. It will rest with individual institutions and the attitudes and integrity of the ODL human resources to continuously improve the quality ODL provision and the capability to adapt and effectively respond to disruptions such as the COVID-19 pandemic and further future challenges. The post-COVID-19 pandemic would have taught important lessons on the significant new roles of online learning and how to continuously improve them in the new landscape of online higher education.

Figures

Appendix List of people consulted through focus group discussion and interviews at UPOU Philippines

References

AAOU (2019), "AAOU quality assurance framework", http://aaou.upou.edu.ph/quality-assurance-framework/ ( ).

(2018), "Indonesia good practices on quality assurance, recognition and student mobility programs", , https://www.asem-education.org/documents/senior-official-meetings-and-ministerial-meetings/isom-jakarta-2018/290-plenary-session-1-intan-ahmad/file ( ).

(2016), "Issues and challenges in open and distance e-learning: perspectives from the Philippines", International Review of Research in Open and Distance Learning, Vol. 17, doi: 10.19173/irrodl.v17i2.1913.

(2015), "An overview of the world of MOOCs", Procedia - Social and Behavioral Sciences, Vol. 174, pp. 427-433, doi: 10.1016/j.sbspro.2015.01.685.

(2019), "Plans, Possibilities and Progress: UP academics moving into the 'next normal'", https://www.up.edu.ph/plans-possibilities-and-progress-up-academics-moving-into-thenext-normal/ ( ).

(2017), "Design and development of online student support system", Pakistan Journal of Distance and Online Learning, Vol. 3 No. 1, pp. 1-8.

University of the Philippines (2020), "University History Emerging excellence: UP in the past 112 years", https://www.up.edu.ph/about-up/university-history/ ( ).

(2013), A Guide to Quality in Online Learning, Academic Partnerships, Mountainview, California, https://www.academicpartnerships.com/Resource/documents/A-Guide-to-Quality-in-Online-Learning.pdf ( ).

Further reading

Acknowledgements

This paper is the outcome of a study on QA in ODL supported by the AAOU Exchange Fellowship at University of the Philippines Open University (UPOU) during the month of November 2019. The authors specifically thank the Chancellor and Vice Chancellor for Academic Affairs of UPOU for hosting the AAOU Fellowship 2019, the Vice-Chancellor of Allama Iqbal Open University (AIOU) Pakistan for including AIOU in the study, and the Rector of Universitas Terbuka (UT) Indonesia for the recommendation and permission to take up the prestigious AAOU Award. Further thanks go to QA Research Assistants of UPOU for their special effort in data collection for this research. More appreciations go to leaders, managers and staff members for the lively discussions and feedback and for sharing ideas on QA in ODL in the three Asian Open Universities.

Corresponding author

# data[].attributes.attachment_link.title
Implementing quality assurance system for open and distance learning in three Asian open universities: Philippines, Indonesia and Pakistan

# data[].attributes.attachment_link.description
Copyright Published in Asian Association of Open Universities Journal. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors.

# data[].attributes.attachment_link.readable_text
August 4, 2020 Posted by Anurag Singh Bug tracking, Case Study, Design, E-Learning, Process, Project Management, Testing, User Acceptance Testing, User Experience 0 thoughts on "eLearning Quality Assurance Checklist with use cases and examples"

Tags: Elearning design and development, Elearning QA, elearning quality, engaging online courses, free elearning testing template, free templates, online course quality, quality checklist

Get your design reviews faster with zipBoard

Get Started Today

Privacy & Cookies: We use cookies to ensure a seamless experience. By continuing to use this website, you agree to the use of the cookies. To find out more, you can access our Privacy Policy. ACCEPT

# data[].attributes.attachment_link.title
eLearning Quality Assurance Checklist with use cases and examples - zipBoard

# data[].attributes.attachment_link.description
August 4, 2020 Posted by Anurag Singh Bug tracking, Case Study, Design, E-Learning, Process, Project Management, Testing, User Acceptance Testing, User Experience 0 thoughts on "eLearning Quality Assurance Checklist with use cases and examples" Tags: Elearning design and development, Elearning QA, elearning quality, engaging online courses, free elearning testing template, free templates, online course quality, quality checklist Get your design reviews faster with zipBoard Get Started Today Privacy & Cookies: We use cookies to ensure a seamless experience.

# data[].attributes.headline
Building and implementation of QA model for eLearning

# data[].attributes.subject
Building and implementation of QA model for eLearning

# data[].attributes.headline
Quality model in online and open education around the globe

# data[].attributes.subject
Quality model in online and open education around the globe

# data[].attributes.attachment_link.readable_text
Effectively evaluating online learning programs starts with two simple ideas: Apply tried-and-true approaches, and take advantage of the opportunity to use new ones. In practice, while there are many effective approaches to evaluating online learning programs, many organizations do not know these approaches or how to apply them.

A theory based on cognitive science research helps explains why. Frame theory holds that "frames are mental structures that shape the way we see the world," according to George Lakoff, professor of Cognitive Science and Linguistics at the University of California at Berkeley [ 1]. As Lakoff notes, frames or framesets shape goals, plans, actions, and perceived quality of outcomes because they strongly influence which facts will be seen (those that fit the frame) and which will be ignored (those that don't).

To understand the power of frames, consider how the cliché expression "Sage on the Stage vs. Guide on the Side" reflects two very different ways of perceiving appropriate instructional practice. Such commonly used frames affect not only how one perceives online learning, but how one evaluates it. For instance, evaluating such factors as social presence, communities of inquiry, or discussion-thread design is far more likely to result from using a frame which views online learners as "connected" rather than "isolated," since the latter frame excludes these factors by definition (i.e., "isolated" learners are not social, do not have communities, and do not participate in discussions).

Ineffective evaluation of online learning results from faulty frames based on misperceptions about both online learning and evaluation itself. Because the frames one uses affect how online learning is perceived, effective approaches to evaluating online learning programs require using appropriate frames. Here is a quick guide to selecting useful frames and avoiding defective ones.

Online learning is not a creature from a different universe. A lot of familiar evaluation tools, for example the commonly-used Kirkpatrick and Phillips scales, apply just as well to online learning programs as to other types of programs. Academic programs routinely measure student satisfaction (Level 1) and learning effectiveness (Levels 1 and 2 on the K/P scales) in online courses, while some corporate and academic online learning programs make attempts to evaluate impact at higher levels (behavior change, effects on business, return on investment). Collections of available evaluation tools, resources, and publications such as The Evaluation Center at Western Michigan University are also useful for evaluating online learning programs.

This faulty frame is more common in academia. In Barbara Wright's view, judgmental evaluation in higher education is a legacy of the accountability movement starting in the 1980s. In this view, "'evaluation' is about making value judgments" and enforcing "accountability" (often in the form of negative consequences) based on evaluation results. Evaluators are generally "external" to the process and thus supposedly "objective," despite their focus on passing judgment. In practice, evaluators are usually resisted or sometimes co-opted in such situations. Seeking to avoid the consequences of negative findings, program administrators focus on damage control, and the process disconnects ownership of implementing proposed recommendations for improvement [ 2]. Approaching online learning evaluation as an occasion for judgment is unnecessary and likely to be unproductive.

A good antidote to the "evaluation as judgment" frame is to consider what evaluation literally means: to draw out ("e-") the worth (from the Latin valere) of something. This means focusing on figuring out what's of value to key "internal" and "external" stakeholders-everyone from project or program participants and their constituents to (in some cases) society at large. Thus, evaluators are in one sense "meaning makers," which is a particularly rewarding aspect of the work. Far from being some abstract philosophical construct, "meaning making" moves beyond data collection, analysis, and results reporting to evaluation, which helps explain the significance and ramifications of program results in terms that make sense to stakeholders. Clients seek and welcome such an approach once they understand it.

During a recent phone conversation, a colleague described her difficulties in a research and evaluation course she was taking for her doctoral program. She kept wanting to describe how the evaluation results could be applied, but the professor kept telling her to focus on the numbers and not be concerned with possible applications. Her efforts to toe this line were passable-she got a "B" in the course. In the world of practice, however, integrating evaluation with strategic planning, dissemination, and sustainability planning makes a lot more sense than treating evaluation as a "standalone" or episodic process.

In instructional design, such integration is taken for granted in models such as ADDIE (the "E" stands for Evaluation) and other Instructional Systems Design (ISD) approaches. Evaluating online learning programs should be integrated into strategic processes as well. This opens up evaluation to a broader range of possibilities for purposeful uses, for instance using evaluations to inform future grant proposals, to justify investment expenditures or funding increases, or to drive continuous quality improvement processes.

At the course level, relying exclusively on end-of-course evaluations is what Matt Champagne of IOTA Solutions refers to as "evaluation as autopsy." Instead of waiting until the patient has died, why not identify and address potential problems before the course is over? For example, David Sachs of Pace University uses ongoing evaluation to monitor online courses offered for the National Coalition for Telecommunications Education & Learning (NACTEL) project. These 15-week courses are evaluated at five-week intervals (Weeks 5, 10, 15) through detailed student surveys. Sachs can quickly review quantitative and narrative student feedback to identify potential problems and alert instructors to deal with them.

Ongoing evaluation can easily be applied to shorter courses as well. For instance, a ten-day online workshop which I recently co-facilitated included a mid-workshop "critique and comments" discussion forum which invited participants to give feedback on activities conducted during the first half of the workshop. We used the feedback to make minor but helpful adjustments to the course.

Sometimes a corpse is all one has to work with, but use ongoing evaluation where possible-when done well, it enhances courses, engages learners, and improves the entire process.

Too often, vendors equate learning with content-in effect, they view learning as solely a function of learner-content interaction. An evaluation approach based on this frame would naturally focus on the efficacy of this interaction-how good is the content? How well do learners interact with it?

Recently, I attended a vendor demonstration of a learning management system (LMS) at the invitation of a client. The LMS had all sorts of features to support development of content and enable it to be packaged as re-usable learning objects (RLOs) in a variety of modes-print, SCORM, HTML, and XML among others. At one point, the vendor rep stated that with their LMS, "you can take a PowerPoint presentation and convert it into a course."

Of course, this was precisely the problem, since my client's interest was in finding an LMS that would support instructor-facilitated, case-based, and other learning approaches. Collaborative group projects, discussion forums, or qualitative assessments did not fit neatly into a "content" box, so the vendor's product was essentially incapable of supporting this functionality coherently.

The real problem with the frame is that learning does not equate with content. Effective learning environments tend to be systems, and in practice evaluating the results or products of a particular program usually involves evaluating other deliverables besides content.

However, evaluating products and results is not enough; evaluating process is often at least as valuable. Listing key evaluation questions which pertain to specific process as well as product deliverables is one common strategy. Another good strategy is to include creating a process guide as a project deliverable. One good tool to use to capture process is to have key project staff complete process or reflection logs, a simple journaling process which invites them to reflect on and record observations about project activities while in progress.

Evaluating process is also important for capturing unanticipated benefits of projects. For instance, the Quality Matters (QM) project, funded by the US Department of Education's Fund for the Improvement of Post-Secondary Education (FIPSE) and administered by MarylandOnline (MOL), was originally focused on facilitating course-sharing among MOL member institutions to improve student access to online learning opportunities. In practice, an unanticipated benefit of the QM project is that the QM process is serving as a viable structure for promoting faculty development. Faculty participants in the peer review process (both peer reviewers and course developers) commonly report that it is a rewarding professional development opportunity.

Paying attention to process is also useful in uncovering significant events which would otherwise be ignored. Recently, a client had delayed project evaluation of a series of online learning modules because only one module serving only a few students had been offered to date, while the other modules were still in production due to development delays. Nonetheless, the client was eager to start a comparative evaluation of this module with their in-person classroom offerings because the window of opportunity was shrinking: They were already planning to make changes to their classroom offerings based on what they had learned from online course development. Rather than waiting for the results and analysis of the pilot offerings of all the modules, what they had learned from the development process was by itself enough evidence for them to make the change. As a result, documenting how this process occurred will become a focus of the project evaluation, but its significance would never have been uncovered without conscious attention to evaluating process.

Comparisons of some sort are unavoidable when evaluating education or training, but comparing delivery modes (in particular online/distance learning with traditional classroom-based instruction) for the purpose of establishing the superiority of one delivery mode over another is specious, irrelevant, and counterproductive [ 3]. Such comparisons assume a non-existent uniformity of practice, so even if a "significant difference" is found, there are a multitude of reasons for that difference which are independent of delivery mode.

A better frame is to focus on making programs better. This may seem too commonsensical to be worth mentioning-until one considers how commonly it is not applied in practice. In some cases, the "evaluation as judgment" frame also gets in the way. By contrast, Wright's "assessment loop" model focuses on using ongoing assessment to support improvement instead of enforcing accountability. The QM peer review process is another model which emphasizes course improvement, in this case for course design. The process is designed to be collegial rather than adversarial, and the labels used to evaluate quality are "meets expectations" or "needs improvement" rather than the more punitive pass/fail frame which is so deeply wired into the brains of educators. Administrators are cautioned against using peer review results judgmentally as part of faculty evaluation, unless it's to reward faculty for participating in the process.

Sample Approaches

A number of established approaches can be easily applied to evaluating online learning programs effectively-for instance ROI (Return on Investment)-focused methods, surveys of student engagement, and many others. However, evaluating online learning programs effectively can be much more than "do what you've always done." Just as the advent of online learning has given practitioners the opportunity to reflect and re-think how learning, training, and instructing are done, the same holds true for evaluating online learning. There are now a variety of new approaches for evaluating online learning effectively. What follows are a few examples of how such approaches can apply productive frames to effective evaluation.

Sloan-C Quality Framework. The Sloan Consortium (Sloan-C) has developed a framework for quality online education [ 4], a composite framework whose five key "pillars" or components (Access, Student Satisfaction, Learning Effectiveness, Faculty Satisfaction, and Cost Effectiveness) can be used separately or in combination with a variety of measures to evaluate the effectiveness of online learning. Although a major focus of this conceptual framework is supporting online learning programs that are equal or better in quality to campus-based programs in higher education, all of the framework's five pillars also focus on effectiveness, and the Faculty Satisfaction and Cost Effectiveness also apply to quality improvement. The Sloan-C Quality Framework also easily accommodates evaluating results and process and integrates well with other evaluation tools; for example, the Kirkpatrick/Phillips (K/P) scales correspond closely with Student Satisfaction (K/P Level 1), Learning Effectiveness (K/P Level 2), and Cost Effectiveness (K Level 4, P Levels 4 & 5).

The CEIT Model for Evaluating Online Learning. Another approach to evaluate online learning programs is a four-stage model I developed for evaluating online learning called the CEIT model [ 5] which focuses on Comparisons, Effectiveness, Quality Improvement, and Transformation.

Using comparisons to understand the key attributes or capabilities of various online learning practices can be useful, so long as one avoids the comparison trap described previously. The results of comparative studies can be used to improve programs as described below.

Focusing on effectiveness can be deceptively simple ("Is it effective?" "What elements make it effective?" "What changes would make it more effective?"), but it enables evaluation of online learning on its own terms and can provide useful information on what works and doesn't work.

Evaluating quality improvement focuses on measuring the effects of implementing changes, so the "evaluating product and process" frame is particularly applicable here since both influence the results in practice. For instance, online learning object repositories often don't improve quality despite containing excellent resources because existing processes don't adequately support their use (good product, bad process). Conversely, online courses often link to a variety of learner support services, but the services themselves may not be adequate (good process, bad product).

Transformation focuses on how online learning programs affect the institution or organization. Although even effective, ever-improving online learning programs often remain isolated islands, some institutions decide to make a conscious and relatively comprehensive effort to create a transformative effect as a result of their experience.

The CEIT model accommodates multiple paths of progression through the four stages. Many online programs start at the comparative stage and progress more or less linearly through the other stages, while others exhibit these stages in a non-linear or concurrent fashion. Although still under development, the CEIT model accommodates application of well-established evaluation tools, determining what's valuable about a project's results and process, and focusing on the ultimate aims of improvement and transformation (making things better).

Mix and Match. These two approaches can be incorporated into an evaluation design; in fact, a "mix and match" approach can be used quite effectively to evaluate online learning. For instance, one evaluation of a series of instructor-led online bioterrorism courses incorporated student characteristics [ 6], applied media attribute theory [ 7], and social presence theory [ 8] into the measurement instruments (student and instructor surveys). Each course module gave students the opportunity to provide feedback on that module; course discussions were also analyzed using a content analysis instrument [ 9] to assess the effectiveness of creating social presence in course discussions. The evaluation data was analyzed and the results used to incorporate improvements into subsequent iterations of the course. [ 3]

Conclusion

The best strategy for effectively evaluating online learning programs is using what you already know, but also to move beyond that-seek to add value and make meaning, integrate evaluation into larger processes, evaluate process as well as results, and keep the focus on making things better. In the short run, clients are happier with results that reflect what they value; in the long run, evaluations that add value also build capacity and increase the potential for integrating evaluation more systemically into the organizational culture. Make the evaluation process more rewarding for all parties concerned by using frames and approaches that matter.

References

1. Lakoff, G. (2004). Don't Think of an Elephant! Know Your Values and Frame the Debate. White River Junction, VT: Chelsea Green, pp. xv, 37.

2. Wright, B. (2004). More Art Than Science: The Post-Secondary Assessment Movement Today. In J. Bourne & J. Moore (Eds.), Elements of Quality Online Education: Into the Mainstream, pp.185-198. Needham, MA: Sloan Center for OnLine Education.

3. Sener, J. (2004/2005). Escaping the Comparison Trap: Evaluating Online Learning on Its Own Terms. Innovate, 1(2). Retrieved June 1, 2005 at: http://innovateonline.info/index.php?view=article&id=11

4. Moore, J. (2002). Elements of Quality: The Sloan-C Framework. Needham, MA: Sloan Center for OnLine Education.

5. Sener, J. (2005). Beyond the Comparison Trap: Strategies for Effective Online Learning Evaluation. Eleventh Sloan-C International Conference on Asynchronous Learning Networks, Orlando, November 2005.

6. Diaz, D. (2000, March/April). Carving a new path for distance education research. The Technology Source. Retrieved November 22, 2005, from http://technologysource.org/article/carving_a_new_path_for_distance_education_research/

7. Smith, P.L., and Dillon, C.L. (1999) Comparing distance learning and classroom learning: conceptual considerations. The American Journal of Distance Education, 13(2), 6-23.

8. Gunawardena, C. (1995). Social presence theory and implications for interaction and collaborative learning in computer conferencing. International Journal of Educational Telecommunications, 1(2-3), 147-166.

9. Swan, K. (2002). Immediacy, social presence, and asynchronous discussion. In J. Bourne & J. C. Moore (Eds) Elements of Quality Online Education, Volume 3. Needham, MA: Sloan Center for Online Education. Accessed online July 17, 2006 at: http://www.kent.edu/rcet/Publications/upload/ISP&ADpict.pdf

# data[].attributes.attachment_link.title
elearn Magazine: Effectively evaluating online learning programs

# data[].attributes.attachment_link.description
eLearn Magazine is a leading source of high-quality information on the uses of online learning and training strategies in a variety of contexts for K-12, higher education, and the corporate workforce.eLearn Magazine presents new technologies and approaches for creating, delivering, and supporting online instruction and workplace performance.

# data[].attributes.attachment_link.readable_text
Review

¹ Global Digital Health Unit, Imperial College London, London, United Kingdom

² Digitally Enabled PrevenTative Health Research Group, Department of Paediatrics, University of Oxford, Oxford, United Kingdom

³ Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Stockholm, Sweden

Corresponding Author

Corresponding Author:

Edward Meinert, MA, MSc, MBA, MPA, PhD

Digitally Enabled PrevenTative Health Research Group

Department of Paediatrics

University of Oxford

John Radcliffe Hospital

Children's Hospital

Oxford, OX3 9DU

United Kingdom

Phone: 44 7824446808

Email: e.meinert14@imperial.ac.uk

ABSTRACT

Background: Massive open online courses (MOOCs) have the potential to make a broader educational impact because many learners undertake these courses. Despite their reach, there is a lack of knowledge about which methods are used for evaluating these courses.

Objective: The aim of this review was to identify current MOOC evaluation methods to inform future study designs.

Methods: We systematically searched the following databases for studies published from January 2008 to October 2018: (1) Scopus, (2) Education Resources Information Center, (3) IEEE (Institute of Electrical and Electronic Engineers) Xplore, (4) PubMed, (5) Web of Science, (6) British Education Index, and (7) Google Scholar search engine. Two reviewers independently screened the abstracts and titles of the studies. Published studies in the English language that evaluated MOOCs were included. The study design of the evaluations, the underlying motivation for the evaluation studies, data collection, and data analysis methods were quantitatively and qualitatively analyzed. The quality of the included studies was appraised using the Cochrane Collaboration Risk of Bias Tool for randomized controlled trials (RCTs) and the National Institutes of Health-National Heart, Lung, and Blood Institute quality assessment tool for cohort observational studies and for before-after (pre-post) studies with no control group.

Results: The initial search resulted in 3275 studies, and 33 eligible studies were included in this review. In total, 16 studies used a quantitative study design, 11 used a qualitative design, and 6 used a mixed methods study design. In all, 16 studies evaluated learner characteristics and behavior, and 20 studies evaluated learning outcomes and experiences. A total of 12 studies used 1 data source, 11 used 2 data sources, 7 used 3 data sources, 4 used 2 data sources, and 1 used 5 data sources. Overall, 3 studies used more than 3 data sources in their evaluation. In terms of the data analysis methods, quantitative methods were most prominent with descriptive and inferential statistics, which were the top 2 preferred methods. In all, 26 studies with a cross-sectional design had a low-quality assessment, whereas RCTs and quasi-experimental studies received a high-quality assessment.

Conclusions: The MOOC evaluation data collection and data analysis methods should be determined carefully on the basis of the aim of the evaluation. The MOOC evaluations are subject to bias, which could be reduced using pre-MOOC measures for comparison or by controlling for confounding variables. Future MOOC evaluations should consider using more diverse data sources and data analysis methods.

International Registered Report Identifier (IRRID): RR2-10.2196/12087

J Med Internet Res 2020;22(4):e13851

doi:10.2196/13851

Citation

Please cite as:

Loading authors...

KEYWORDS

Introduction

Massive open online courses (MOOCs) are free Web-based open courses available to anyone everywhere and have the potential to revolutionize education by increasing the accessibility and reach of education to large numbers of people []. However, questions remain regarding the quality of education provided through MOOCs []. One way to ensure the quality of MOOCs is through the evaluation of the course in a systematic way with the goal of improvement over time []. Although research about MOOCs has increased in recent years, there is limited research on the evaluation of MOOCs []. In addition, there is a need for effective evaluation methods for appraising the effectiveness and success of the courses.

Despite the limitations in MOOC evaluation methods, several reviews have been conducted on MOOC-related research methods, without specifically focusing on MOOC evaluations. Two recent systematic reviews were published synthesizing MOOC research methods and topics [,]. Zhu et al [] and Bozkurt et al [] recommended further research on the methodological approaches for MOOC evaluation. This research found little focus on the quality of the techniques and methodologies used []. In addition, a large number of studies on MOOCs examine general pedagogical aspects of the course without evaluating the course itself. Although the general evaluation of MOOC education and pedagogy is useful, it is essential that courses are also evaluated [].

To address the gaps in MOOC evaluation methods in the literature, this systematic review aimed to identify and analyze current MOOC evaluation methods. The objective of this review was to inform future MOOC evaluation methodology.

Methods

This review explored the following research question: What methods have been used to evaluate MOOCs? []. This systematic review was conducted according to the Cochrane guidelines [] and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines () []. As the review only used publicly available information, an ethics review board approval was not required. The review was executed in accordance with the protocol published by Foley et al [].

Eligibility Criteria

Eligible studies focused on the evaluation of MOOCs with reference to the course design, materials, or topics. The evaluation used the following population, intervention, comparator, outcome (PICO) framework for inclusion in the study:

Population: learners in any geographic area who have participated in MOOCs [].

Intervention: MOOC evaluation methods. This is intended to be broad to include qualitative, quantitative, and mixed methods [].

Comparator: studies did not need to include a comparator for inclusion in this systematic review [].

Outcome: learner-focused outcomes such as attitudes, cognitive changes, learner satisfaction, etc, will be assessed [].

Further to the abovementioned PICO framework, we used the following inclusion and exclusion criteria.

Inclusion Criteria

Studies with a primary focus on MOOC evaluation and studies that have applied or reviewed MOOC evaluation methods (quantitative, qualitative, or mixed methods) [].

Studies published from 2008 to 2018 [].

All types of MOOCs, for example, extended MOOCs, connectivist MOOCs, language MOOCs, or hybrid MOOCs.

Exclusion Criteria

Studies not in the English language [].

Studies that primarily focused on e-learning or blended learning instead of MOOCs [].

Studies that focused only on understanding MOOC learners such as their behaviors or motivation to join MOOCs, without referring to the MOOC.

Studies that focused on machine learning or predictive models to predict learner behavior.

Search Strategy

We searched the following databases for potentially relevant literature from January 2008 to October 2018: (1) Scopus, (2) Education Resources Information Center, (3) IEEE (Institute of Electrical and Electronic Engineers) Xplore, (4) Medical Literature Analysis and Retrieval System Online/PubMed, (5) Web of Science, (6) British Education Index, and (7) Google Scholar search engine. The first search was performed in Scopus. The search words and terms for Scopus were as follows: (mooc* OR "massive open online course" OR coursera OR edx OR odl OR udacity OR futurelearn AND evaluat* OR measur* OR compar* OR analys* OR report* OR assess* AND knowledge OR "applicable knowledge" OR retent* OR impact OR quality OR improv* OR environment OR effect "learning outcome" OR learning). The asterisks after the search terms allow all terms beginning with the same root word to be included in the search. The search terms were then adjusted for each database. The complete search strategy for each database can be found in the protocol by Foley et al [] and in . In addition, we scanned the reference lists of included studies.

Selection of Studies

Two reviewers (AA and CL) independently screened the titles and abstracts of the articles for eligibility. Selected studies were identified for full-text reading. Disagreements between the reviewers were resolved by discussions with a third reviewer (EM). Few studies (<10) were discussed with a third reviewer.

Data Extraction

The following information was extracted from each included study using a data abstraction form (): (1) article title, country of the first author, and year of publication; (2) study aims; (3) evaluation: evaluation method, study design, evaluation type (evaluation of a single MOOC, multiple MOOCs, or review of a method), data collection methods, data analysis methods, and number of participants; and (4) outcome measures of the study: learner-focused outcomes and other outcomes. The studies were classified as quantitative, mixed methods, or qualitative based on the methods used. Studies were considered as mixed methods if they used a combination of qualitative or quantitative techniques, methods, approaches, concepts, or language in the same study [].

Assessment of Methodological Quality

The Cochrane Collaboration Risk of Bias Tool for randomized controlled trials (RCTs) [] and the National Institutes of Health-National Heart, Lung, and Blood Institute quality assessment tool for cohort observational studies and for before-after (pre-post) studies with no control group [] were used to assess the methodological quality of the included studies depending on their study design.

Data Synthesis

We summarized the data graphically and descriptively. The evaluation results were reported according to the design thinking approach for evaluations that follows the subsequent order: (1) problem framing, (2) data collection, (3) analysis, and (4) interpretation [].

Problem Framing

The evaluation-focused categories in the problem framing section were determined through discussions among the primary authors to summarize study aims and objectives. The 3 categories used in the evaluation-focused categories were defined as follows:

1. The learner-focused evaluation seeks to gain insight into the learner characteristics and behavior, including metrics such as completion and participation rates, satisfaction rates, their learning experiences, and outcomes.

2. Teaching-focused evaluation studies aim to analyze pedagogical practices so as to improve teaching.

3. MOOC-focused evaluation studies aim to better understand the efficacy of the learning platform to improve the overall impact of these courses.

Further to the evaluation-focused categories, the subcategories were generated by conducting a thematic analysis of the MOOC evaluation studies' aims and objectives. The themes resulted through an iterative process where study aims were coded and then consolidated into themes by the first author. The themes were then discussed with and reviewed by the second author until an agreement was reached.

Data Collection Analysis and Interpretation

The categories reported in the data collection sections were all representations of what the studies reported to be the data collection method. The categorization of the learner-focused parameters was done based on how the authors identified the outcomes. For example, if authors mention that the reported outcome was measuring learners' attitudes to evaluate overall MOOC experience, the parameter was recorded in the learner experience category. Similarly, if the authors mentioned that the reported outcome was evaluating what students gained from the course, the parameter was recorded as longer term learner outcomes.

Results

In this section, we have described the search results and the methodological quality assessment results. We have then described the study findings using the following categories for MOOC evaluation: research design, aim, data collection methods, data analysis methods, and analysis and interpretation.

Search Results

There were 3275 records identified in the literature search and 2499 records remained after duplicates were removed. Records were screened twice before full-text reading. In the first screening (n=2499), all articles that did not focus on MOOCs specifically were removed (). In the second screening (n=906), all articles that did not focus on MOOC learners or MOOC evaluation methods were removed (). This was followed by full-text reading of 154 studies (). An additional 5 studies were identified by searching the bibliographies of the included studies. In total, 33 publications were included in this review. There were 31 cross-sectional studies, 1 randomized trial, and 1 quasi-experimental study. The completed data abstraction forms of the included studies are in .

‎
Figure 1. A Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart of the literature search.
View this figure

Methodological Quality

The RCT included in this study [] received a low risk-of-bias classification ().

A quality assessment of the quasi-experimental study using longitudinal pretests and posttests [] is included in .

Massive Open Online Course Evaluation Research Design

In total, 6 studies used a comparator in their methods. A study compared precourse and postcourse surveys by performing a chi-square test of changes in confidence, attitudes, and knowledge []. A study compared the average assignment and final essay scores of MOOC learners with face-to-face learners and calculated 2 independent sample t tests to compare the differences between learners but did not include any pre- and posttest or survey results []. In all, 4 studies conducted pretest and posttest analyses [,]. Hossain et al [] used an RCT design and calculated the mean between-group differences of knowledge, confidence, and satisfaction comparing MOOC learners with other Web-based learners. Colvin et al [] calculated normalized gain using item response between pretest and posttest scores and the Item Response Theory for weekly performance compared with that of on-campus learners. Rubio et al [] compared the pretest mean and posttest mean of comprehensibility scores in a MOOC, comparing results with those of face-to-face learners []. Konstan et al [] calculated knowledge test gains by performing a paired t test of average knowledge gains, comparing these gains with those of face-to-face learners and (comparing 2 learner groups) the average normalized learning gains among all learners [].

Aim of Massive Open Online Course Evaluations

The aim or objective of MOOC evaluations included in this review can be categorized into learner-focused, teaching-focused, and MOOC-focused evaluation aims (). In all, 16 studies evaluated learner characteristics and behavior and 20 studies evaluated learning outcomes and experiences. One of the least studied aspects of MOOC evaluation is pedagogical practices, which were only evaluated by 2 studies [,].

Massive Open Online Course Evaluation Data Collection Methods

Massive Open Online Course Evaluation Analysis and Interpretation

In terms of the data analysis methods, quantitative methods were the only type of method used in 16 studies with descriptive and inferential statistics, the top 2 preferred methods. Qualitative analysis methods such as thematic analyses, which can include grounded theory [], focused coding [,], and content analysis [,], were mainly used in qualitative studies.

A summary of the parameters, indicators, and data analysis used for the MOOC evaluation can be found in . Most notably, inferential statistics were used to analyze learning outcomes () such as the comparison of means or the use of regression methods to analyze quiz or test grades. These outcomes were also used as a measure to evaluate the overall effectiveness of a MOOC by the studies. shows how the data collection method uses mentioned in were measured and analyzed. In general, studies focused on measuring learner engagement and learners' behavior-related indicators. Studies referred to learning in different ways such as learning, learning performance, learning outcome, or gain in comprehensibility depending on the learning material of the course. Other studies considered learning outcomes such as knowledge retention or what students took away from the course. There was a consensus that learner engagement can be measured by measuring the various learner activities in the course, whereas learner behavior was a more general term used by studies to describe the different MOOC evaluation measures. For teaching-focused evaluation, both Mackness et al [] and Singh et al [] used learner parameters to reflect and analyze pedagogical practices.

Discussion

This study aimed to review current MOOC evaluation methods to understand the methods that have been used in published MOOC studies and subsequently to inform future designs of MOOC evaluation methods. Owing to the diversity of MOOC topics and learners, it is not possible to propose a single evaluation method for all MOOCs. Researchers aiming to evaluate a MOOC should choose a method based on the aims of their evaluation or the parameters they would like to measure. In general, data collection methods were similar in most evaluations, such as the use of interviews or survey data, and the analysis methods were highly heterogeneous among studies.

Massive Open Online Course Evaluation Research Design

The cross-sectional study design was used in 31 of 33 of the included studies. The cross-sectional study design was used when the aim was to investigate the factors affecting outcomes for a population at a given time point []. For the MOOC evaluation, this is particularly useful for observing the population of learners and for understanding the factors affecting the success and impact of a MOOC. They are relatively inexpensive to conduct and can assess many outcomes and factors at the same time. However, cross-sectional study designs are subject to nonresponse bias, which means that studies are only representative of those who participated, who incidentally may happen to be different from the rest of the population [].

One of the most effective methods of evaluation used in MOOCs was the use of baseline data to compare outcomes. Studies that did pretests and posttests had a less likelihood of bias in their outcomes owing to the measurement of exposure before the measurement of outcome []. Even when studies used pre- and postcourse surveys or tests, they were not longitudinal in design, as such a design requires a follow-up of the same individuals and requires observing them at multiple time points []. Therefore, the use of pre- and postsurveys or tests without linking the individuals may simply represent a difference in the groups studied rather than changes in learning or learner outcomes. The advantages of this method are that it can reduce bias, and quasi-experimental studies are known as strong methods. However, the disadvantage is that although this method may work with assessing learning, such as memorizing information, it may not work to assess skill development or the application of skills.

Aim of Massive Open Online Course Evaluations

Understanding the aim behind the evaluation of MOOCs is critically important in designing MOOC evaluation methods as it influences the performance indicators and parameters to be evaluated. More importantly, motivation for the evaluation determines the data methods that will be used. One reason for the inability to conclude a standardized evaluation method from this review is that studies differ in the aspects and purposes of why they are conducting the evaluation. For example, not all studies perform evaluations of MOOCs to evaluate overall effectiveness, which is an important aspect to consider if MOOCs are to be adopted more formally in higher education []. The variability in the motivation of MOOC evaluations may also explain the high variability in the outcomes measured and reported.

Data Collection Methodology

In all, 12 studies used 1 data source and 11 studies used 2 data sources (), which is not different from previous findings []. The results of this study also show that there is high flexibility in data collection methods for MOOC evaluations from survey data to LMS data to more distinct methods such as online focus groups []. The number of participants in the studies was exceedingly varied. This is due to the difference in the data collection methods used. For example, studies with data captured through the LMS, which is capable of capturing data from all of the learners who joined the course, had the highest number of learners. On the contrary, studies that used more time-consuming methods, such as surveys or interviews, generally had a lower number of participants. It is important to note that the MOOC evaluation is not necessarily improved by increasing the number of data sources but rather by conducting a meaningful analysis of the available data. Some studies preferred multiple methods of evaluation and assessment of learning. One paper argued that this allows to evaluate learning of the diverse MOOC population in a more effective way []. Studies should use the best data collection methods to answer their research aims and questions.

Analysis and Interpretation

In total, 16 of 33 studies used only quantitative methods for analysis (), which is in line with the general MOOC research, which has been predominated by quantitative methods [,]. Studies used statistical methods such as descriptive and inferential statistics for data analysis and interpretation of results. The availability of data from sources such as the LMS may have encouraged the use of descriptive statistical methods []. However, 17 of the 33 included studies used some form of qualitative data analysis methods either by using a qualitative study design or by using a mixed methods study design (). This may be explained by the recent (2016-2017) rise in the use of qualitative methods in MOOC research [].

Although inferential statistics can help create better outcomes from studies, this is not always possible. For example, one study [] mentioned a high variation between pre- and postcourse survey participant numbers and another [] mentioned a small sample size as reasons for not using inferential statistical methods. It should be noted that using data from multiple sources and having a large sample size does not guarantee the quality of the evaluation methods.

In MOOC research, qualitative data can be useful to understand the meaning of different behaviors as quantitative data, oftentimes, cannot answer why things happened [].

Outcome Measures

The outcome measures reported greatly varied among studies, which is expected, as identifying the right outcome measures is an inherent challenge in educational research, including more traditional classroom-based studies [].

The choice of evaluation methods is highly dependent on the aim of the evaluation and the size of the MOOCs. For quantitative measures, such as completion and participation rates, metrics can be easily collected through the MOOC platform. However, these metrics alone may be insufficient to provide insights into why students fail to complete the course for future improvement. Although it may be difficult to represent the problem holistically using qualitative methods, it can be useful in providing insights from individuals who participated in the MOOCs. Mixed methods studies combine the 2 modalities to better understand metrics generated and produce greater insights for future improvement of the MOOCs.

Learning outcomes were mostly analyzed by inferential statistical methods owing to the use of pretest and posttest methods and the calculation of gains in learning. This method may be most suited for MOOCs that require knowledge retention. Learning parameters also involved a lot of comparisons, either a comparison with pre-MOOC measures or a comparison with other learners or both. Social interactions were studied in 2 of the MOOC evaluations using social network analysis methods. Although the MOOC completion rate has been often cited as a parameter for MOOC success, it can be noticed that studies started to move away from only using completion rates. For example, studies looked at completion of different steps of the MOOCs or looked at overall completion. The learning outcomes reported in this review should be used with caution as not all of them have been validated or assessed for their reliability except for a few.

Methodological Quality

In total, 26 studies with a cross-sectional design had a low-quality assessment, whereas RCTs and quasi-experimental studies received a high-quality assessment. Having a high level of bias affects the generalizability of studies, which is a common problem in most research using data from MOOCs [,]. The availability of high risks of bias in current MOOC evaluations requires a closer look at what were the sources of bias and what methods can be used to reduce them. The use of not validated, self-reported data sources and the lack of longitudinal data also increases the risk of bias in these studies []. However, although most MOOCs struggle with learner retention and MOOC completion rates [], it is understandable that studies are not able to collect longitudinal data.

Future Directions

The scarcity of studies focusing on the evaluation of the effectiveness of particular MOOCs relative to the number of available studies on MOOCs raises some questions. For example, many studies that were excluded from this review studied MOOC learners or aspects of the MOOCs without conducting an evaluation of course success or effectiveness. As shown in this review, there is a diverse range of evaluation methods, and the quality of these evaluation studies can be as diverse. The motivation of the evaluation exercise should be the basis of the evaluation study design to design effective quantitative or qualitative data collection strategies. The development of general guidance, standardized performance indicators, and an evaluation framework using a design thinking approach can allow these MOOC evaluation exercises to yield data of better quality and precision and allow improved evaluation outcomes. To provide a comprehensive evaluation of MOOCs, studies should try to use a framework to be able to systematically review all of the aspects of the course.

In general, the adoption of a mixed methods analysis considering both quantitative and qualitative data can be more useful for evaluating the overall quality of MOOCs. Although it is useful to have quantitative data such as learner participation and dropout rates, qualitative data gathered through interviews and opinion mining provide valuable insights into the reasons behind the success or failure of a MOOC. Studies of MOOC evaluations should aim to use data collection and analysis methods that can minimize the risk of bias and provide objective results. Whenever possible, studies should use comparison methods, such as the use of pretest or posttest or a comparison with other types of learners, as a control measure. In addition, learner persistence is an important indicator for MOOC evaluation that needs to be addressed in future research.

Strengths and Limitations

To our knowledge, this is the first study to systematically review the evaluation methods of MOOCs. The findings of this review can serve future MOOC evaluators with recommendations on their evaluation methods to facilitate better study designs and maximize the impact of these Web-based platforms. However, as a lot of MOOCs are not necessarily provided by universities and systematically evaluated and published, the scope of this review can only reflect a small part of MOOC evaluation studies.

Conclusions

There is no one way of completing a MOOC evaluation, but there are considerations that should be taken into account in every evaluation. First, because MOOCs are very large, there is a tendency to use quantitative methods using aggregate-level data. However, aggregate-level data do not always tell why things are happening. Qualitative data could further help interpret the results by exploring why things are happening. Evaluations lacked longitudinal data and very few accounted for confounding variables owing to data collection challenges associated with MOOCs such as not having longitudinal data or not having enough data sources. Future studies could help identify how these challenges could be overcome or minimized.

LMS may not report useful findings on an individual level, but they should still be considered and used in MOOC evaluations. Big data in the form of learning analytics can help with decision making, predicting learner behavior, and providing a more comprehensive picture of the phenomena studied []. Studies should still consider using LMS as it can provide a valuable addition to the research, but researchers need to be careful about the depth of the findings that can be concluded from LMS-only datasets.

The use of qualitative data could help enhance the findings from the studies by explaining the phenomena. Both quantitative and qualitative methods could play a key role in MOOC evaluations.

Current MOOC evaluations are subject to many sources of bias owing to the nature of the courses being open and available to a very large and diverse number of participants. However, methods are available to reduce the sources of bias. Studies could use a comparator, such as pretest scores, or other types of learners to be able to calculate relative changes in learning. In addition, studies could control for confounding variables to reduce bias.

This review has provided an in-depth view of how MOOCs can be evaluated and explored the methodological approaches used. Exploring MOOC methodological approaches has been stated as an area for future research []. The review also provided recommendations for future MOOC evaluations and for future research in this area to help improve the quality and reliability of the studies. MOOC evaluations could contribute to the development and improvement of these courses.

Acknowledgments

The authors would like to thank the medical librarian, Rebecca Jones, for her guidance in the search methods and for reviewing the search strategy used in this protocol. This work was funded by the European Institute of Technology and Innovation Health (Grant No. 18654).

Authors' Contributions

AA and CL completed the screening of articles and data analysis. AA and CL completed the first draft of the manuscript. All authors reviewed and edited the manuscript for content and clarity. EM was the guarantor.

Conflicts of Interest

None declared.
‎

Multimedia Appendix 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2009 checklist.
DOCX File , 19 KB
‎ ‎ ‎

Multimedia Appendix 4

Quality assessment results of the Randomized Controlled Trial [] using the Cochrane Collaboration Risk of Bias Tool.
DOCX File , 14 KB
‎

Multimedia Appendix 5

Quality assessment results of cross-sectional studies using the NIH - National Heart, Lung and Blood Institute quality assessment tool.
DOCX File , 43 KB
‎

Multimedia Appendix 6

Quality assessment results for the quasi experimental study using the Cochrane Collaboration Risk of Bias Tool for Before-After (Pre-Post) Studies With No Control Group.
DOCX File , 15 KB

‎
Abbreviations

IEEE: Institute of Electrical and Electronic Engineers

LMS: learning management system

MOOC: massive open online course

PICO: population, intervention, comparator, outcome

RCT: randomized controlled trial

Edited by G Eysenbach; submitted 27.02.19; peer-reviewed by E Toki, O Las Vergnas, F Sendra-Portero; comments to author 20.03.19; revised version received 20.11.19; accepted 22.01.20; published 27.04.20

©Abrar Alturkistani, Ching Lam, Kimberley Foley, Terese Stenfors, Elizabeth R Blum, Michelle Helena Van Velthoven, Edward Meinert. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 27.04.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.

ABSTRACT

Background: Massive open online courses (MOOCs) have the potential to make a broader educational impact because many learners undertake these courses. Despite their reach, there is a lack of knowledge about which methods are used for evaluating these courses.

Objective: The aim of this review was to identify current MOOC evaluation methods to inform future study designs.

Methods: We systematically searched the following databases for studies published from January 2008 to October 2018: (1) Scopus, (2) Education Resources Information Center, (3) IEEE (Institute of Electrical and Electronic Engineers) Xplore, (4) PubMed, (5) Web of Science, (6) British Education Index, and (7) Google Scholar search engine. Two reviewers independently screened the abstracts and titles of the studies. Published studies in the English language that evaluated MOOCs were included. The study design of the evaluations, the underlying motivation for the evaluation studies, data collection, and data analysis methods were quantitatively and qualitatively analyzed. The quality of the included studies was appraised using the Cochrane Collaboration Risk of Bias Tool for randomized controlled trials (RCTs) and the National Institutes of Health-National Heart, Lung, and Blood Institute quality assessment tool for cohort observational studies and for before-after (pre-post) studies with no control group.

Results: The initial search resulted in 3275 studies, and 33 eligible studies were included in this review. In total, 16 studies used a quantitative study design, 11 used a qualitative design, and 6 used a mixed methods study design. In all, 16 studies evaluated learner characteristics and behavior, and 20 studies evaluated learning outcomes and experiences. A total of 12 studies used 1 data source, 11 used 2 data sources, 7 used 3 data sources, 4 used 2 data sources, and 1 used 5 data sources. Overall, 3 studies used more than 3 data sources in their evaluation. In terms of the data analysis methods, quantitative methods were most prominent with descriptive and inferential statistics, which were the top 2 preferred methods. In all, 26 studies with a cross-sectional design had a low-quality assessment, whereas RCTs and quasi-experimental studies received a high-quality assessment.

Conclusions: The MOOC evaluation data collection and data analysis methods should be determined carefully on the basis of the aim of the evaluation. The MOOC evaluations are subject to bias, which could be reduced using pre-MOOC measures for comparison or by controlling for confounding variables. Future MOOC evaluations should consider using more diverse data sources and data analysis methods.

International Registered Report Identifier (IRRID): RR2-10.2196/12087

J Med Internet Res 2020;22(4):e13851

doi:10.2196/13851

Citation

Please cite as:

Loading authors...

KEYWORDS

Introduction

Massive open online courses (MOOCs) are free Web-based open courses available to anyone everywhere and have the potential to revolutionize education by increasing the accessibility and reach of education to large numbers of people []. However, questions remain regarding the quality of education provided through MOOCs []. One way to ensure the quality of MOOCs is through the evaluation of the course in a systematic way with the goal of improvement over time []. Although research about MOOCs has increased in recent years, there is limited research on the evaluation of MOOCs []. In addition, there is a need for effective evaluation methods for appraising the effectiveness and success of the courses.

Despite the limitations in MOOC evaluation methods, several reviews have been conducted on MOOC-related research methods, without specifically focusing on MOOC evaluations. Two recent systematic reviews were published synthesizing MOOC research methods and topics [,]. Zhu et al [] and Bozkurt et al [] recommended further research on the methodological approaches for MOOC evaluation. This research found little focus on the quality of the techniques and methodologies used []. In addition, a large number of studies on MOOCs examine general pedagogical aspects of the course without evaluating the course itself. Although the general evaluation of MOOC education and pedagogy is useful, it is essential that courses are also evaluated [].

To address the gaps in MOOC evaluation methods in the literature, this systematic review aimed to identify and analyze current MOOC evaluation methods. The objective of this review was to inform future MOOC evaluation methodology.

Methods

This review explored the following research question: What methods have been used to evaluate MOOCs? []. This systematic review was conducted according to the Cochrane guidelines [] and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines () []. As the review only used publicly available information, an ethics review board approval was not required. The review was executed in accordance with the protocol published by Foley et al [].

Eligibility Criteria

Eligible studies focused on the evaluation of MOOCs with reference to the course design, materials, or topics. The evaluation used the following population, intervention, comparator, outcome (PICO) framework for inclusion in the study:

Population: learners in any geographic area who have participated in MOOCs [].

Intervention: MOOC evaluation methods. This is intended to be broad to include qualitative, quantitative, and mixed methods [].

Comparator: studies did not need to include a comparator for inclusion in this systematic review [].

Outcome: learner-focused outcomes such as attitudes, cognitive changes, learner satisfaction, etc, will be assessed [].

Further to the abovementioned PICO framework, we used the following inclusion and exclusion criteria.

Inclusion Criteria

Studies with a primary focus on MOOC evaluation and studies that have applied or reviewed MOOC evaluation methods (quantitative, qualitative, or mixed methods) [].

Studies published from 2008 to 2018 [].

All types of MOOCs, for example, extended MOOCs, connectivist MOOCs, language MOOCs, or hybrid MOOCs.

Exclusion Criteria

Studies not in the English language [].

Studies that primarily focused on e-learning or blended learning instead of MOOCs [].

Studies that focused only on understanding MOOC learners such as their behaviors or motivation to join MOOCs, without referring to the MOOC.

Studies that focused on machine learning or predictive models to predict learner behavior.

Search Strategy

We searched the following databases for potentially relevant literature from January 2008 to October 2018: (1) Scopus, (2) Education Resources Information Center, (3) IEEE (Institute of Electrical and Electronic Engineers) Xplore, (4) Medical Literature Analysis and Retrieval System Online/PubMed, (5) Web of Science, (6) British Education Index, and (7) Google Scholar search engine. The first search was performed in Scopus. The search words and terms for Scopus were as follows: (mooc* OR "massive open online course" OR coursera OR edx OR odl OR udacity OR futurelearn AND evaluat* OR measur* OR compar* OR analys* OR report* OR assess* AND knowledge OR "applicable knowledge" OR retent* OR impact OR quality OR improv* OR environment OR effect "learning outcome" OR learning). The asterisks after the search terms allow all terms beginning with the same root word to be included in the search. The search terms were then adjusted for each database. The complete search strategy for each database can be found in the protocol by Foley et al [] and in . In addition, we scanned the reference lists of included studies.

Selection of Studies

Two reviewers (AA and CL) independently screened the titles and abstracts of the articles for eligibility. Selected studies were identified for full-text reading. Disagreements between the reviewers were resolved by discussions with a third reviewer (EM). Few studies (<10) were discussed with a third reviewer.

Data Extraction

The following information was extracted from each included study using a data abstraction form (): (1) article title, country of the first author, and year of publication; (2) study aims; (3) evaluation: evaluation method, study design, evaluation type (evaluation of a single MOOC, multiple MOOCs, or review of a method), data collection methods, data analysis methods, and number of participants; and (4) outcome measures of the study: learner-focused outcomes and other outcomes. The studies were classified as quantitative, mixed methods, or qualitative based on the methods used. Studies were considered as mixed methods if they used a combination of qualitative or quantitative techniques, methods, approaches, concepts, or language in the same study [].

Assessment of Methodological Quality

The Cochrane Collaboration Risk of Bias Tool for randomized controlled trials (RCTs) [] and the National Institutes of Health-National Heart, Lung, and Blood Institute quality assessment tool for cohort observational studies and for before-after (pre-post) studies with no control group [] were used to assess the methodological quality of the included studies depending on their study design.

Data Synthesis

We summarized the data graphically and descriptively. The evaluation results were reported according to the design thinking approach for evaluations that follows the subsequent order: (1) problem framing, (2) data collection, (3) analysis, and (4) interpretation [].

Problem Framing

The evaluation-focused categories in the problem framing section were determined through discussions among the primary authors to summarize study aims and objectives. The 3 categories used in the evaluation-focused categories were defined as follows:

1. The learner-focused evaluation seeks to gain insight into the learner characteristics and behavior, including metrics such as completion and participation rates, satisfaction rates, their learning experiences, and outcomes.

2. Teaching-focused evaluation studies aim to analyze pedagogical practices so as to improve teaching.

3. MOOC-focused evaluation studies aim to better understand the efficacy of the learning platform to improve the overall impact of these courses.

Further to the evaluation-focused categories, the subcategories were generated by conducting a thematic analysis of the MOOC evaluation studies' aims and objectives. The themes resulted through an iterative process where study aims were coded and then consolidated into themes by the first author. The themes were then discussed with and reviewed by the second author until an agreement was reached.

Data Collection Analysis and Interpretation

The categories reported in the data collection sections were all representations of what the studies reported to be the data collection method. The categorization of the learner-focused parameters was done based on how the authors identified the outcomes. For example, if authors mention that the reported outcome was measuring learners' attitudes to evaluate overall MOOC experience, the parameter was recorded in the learner experience category. Similarly, if the authors mentioned that the reported outcome was evaluating what students gained from the course, the parameter was recorded as longer term learner outcomes.

Results

In this section, we have described the search results and the methodological quality assessment results. We have then described the study findings using the following categories for MOOC evaluation: research design, aim, data collection methods, data analysis methods, and analysis and interpretation.

Search Results

There were 3275 records identified in the literature search and 2499 records remained after duplicates were removed. Records were screened twice before full-text reading. In the first screening (n=2499), all articles that did not focus on MOOCs specifically were removed (). In the second screening (n=906), all articles that did not focus on MOOC learners or MOOC evaluation methods were removed (). This was followed by full-text reading of 154 studies (). An additional 5 studies were identified by searching the bibliographies of the included studies. In total, 33 publications were included in this review. There were 31 cross-sectional studies, 1 randomized trial, and 1 quasi-experimental study. The completed data abstraction forms of the included studies are in .

‎
Figure 1. A Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart of the literature search.
View this figure

Methodological Quality

The RCT included in this study [] received a low risk-of-bias classification ().

A quality assessment of the quasi-experimental study using longitudinal pretests and posttests [] is included in .

Massive Open Online Course Evaluation Research Design

In total, 6 studies used a comparator in their methods. A study compared precourse and postcourse surveys by performing a chi-square test of changes in confidence, attitudes, and knowledge []. A study compared the average assignment and final essay scores of MOOC learners with face-to-face learners and calculated 2 independent sample t tests to compare the differences between learners but did not include any pre- and posttest or survey results []. In all, 4 studies conducted pretest and posttest analyses [,]. Hossain et al [] used an RCT design and calculated the mean between-group differences of knowledge, confidence, and satisfaction comparing MOOC learners with other Web-based learners. Colvin et al [] calculated normalized gain using item response between pretest and posttest scores and the Item Response Theory for weekly performance compared with that of on-campus learners. Rubio et al [] compared the pretest mean and posttest mean of comprehensibility scores in a MOOC, comparing results with those of face-to-face learners []. Konstan et al [] calculated knowledge test gains by performing a paired t test of average knowledge gains, comparing these gains with those of face-to-face learners and (comparing 2 learner groups) the average normalized learning gains among all learners [].

Aim of Massive Open Online Course Evaluations

The aim or objective of MOOC evaluations included in this review can be categorized into learner-focused, teaching-focused, and MOOC-focused evaluation aims (). In all, 16 studies evaluated learner characteristics and behavior and 20 studies evaluated learning outcomes and experiences. One of the least studied aspects of MOOC evaluation is pedagogical practices, which were only evaluated by 2 studies [,].

Massive Open Online Course Evaluation Data Collection Methods

Massive Open Online Course Evaluation Analysis and Interpretation

In terms of the data analysis methods, quantitative methods were the only type of method used in 16 studies with descriptive and inferential statistics, the top 2 preferred methods. Qualitative analysis methods such as thematic analyses, which can include grounded theory [], focused coding [,], and content analysis [,], were mainly used in qualitative studies.

A summary of the parameters, indicators, and data analysis used for the MOOC evaluation can be found in . Most notably, inferential statistics were used to analyze learning outcomes () such as the comparison of means or the use of regression methods to analyze quiz or test grades. These outcomes were also used as a measure to evaluate the overall effectiveness of a MOOC by the studies. shows how the data collection method uses mentioned in were measured and analyzed. In general, studies focused on measuring learner engagement and learners' behavior-related indicators. Studies referred to learning in different ways such as learning, learning performance, learning outcome, or gain in comprehensibility depending on the learning material of the course. Other studies considered learning outcomes such as knowledge retention or what students took away from the course. There was a consensus that learner engagement can be measured by measuring the various learner activities in the course, whereas learner behavior was a more general term used by studies to describe the different MOOC evaluation measures. For teaching-focused evaluation, both Mackness et al [] and Singh et al [] used learner parameters to reflect and analyze pedagogical practices.

Discussion

This study aimed to review current MOOC evaluation methods to understand the methods that have been used in published MOOC studies and subsequently to inform future designs of MOOC evaluation methods. Owing to the diversity of MOOC topics and learners, it is not possible to propose a single evaluation method for all MOOCs. Researchers aiming to evaluate a MOOC should choose a method based on the aims of their evaluation or the parameters they would like to measure. In general, data collection methods were similar in most evaluations, such as the use of interviews or survey data, and the analysis methods were highly heterogeneous among studies.

Massive Open Online Course Evaluation Research Design

The cross-sectional study design was used in 31 of 33 of the included studies. The cross-sectional study design was used when the aim was to investigate the factors affecting outcomes for a population at a given time point []. For the MOOC evaluation, this is particularly useful for observing the population of learners and for understanding the factors affecting the success and impact of a MOOC. They are relatively inexpensive to conduct and can assess many outcomes and factors at the same time. However, cross-sectional study designs are subject to nonresponse bias, which means that studies are only representative of those who participated, who incidentally may happen to be different from the rest of the population [].

One of the most effective methods of evaluation used in MOOCs was the use of baseline data to compare outcomes. Studies that did pretests and posttests had a less likelihood of bias in their outcomes owing to the measurement of exposure before the measurement of outcome []. Even when studies used pre- and postcourse surveys or tests, they were not longitudinal in design, as such a design requires a follow-up of the same individuals and requires observing them at multiple time points []. Therefore, the use of pre- and postsurveys or tests without linking the individuals may simply represent a difference in the groups studied rather than changes in learning or learner outcomes. The advantages of this method are that it can reduce bias, and quasi-experimental studies are known as strong methods. However, the disadvantage is that although this method may work with assessing learning, such as memorizing information, it may not work to assess skill development or the application of skills.

Aim of Massive Open Online Course Evaluations

Understanding the aim behind the evaluation of MOOCs is critically important in designing MOOC evaluation methods as it influences the performance indicators and parameters to be evaluated. More importantly, motivation for the evaluation determines the data methods that will be used. One reason for the inability to conclude a standardized evaluation method from this review is that studies differ in the aspects and purposes of why they are conducting the evaluation. For example, not all studies perform evaluations of MOOCs to evaluate overall effectiveness, which is an important aspect to consider if MOOCs are to be adopted more formally in higher education []. The variability in the motivation of MOOC evaluations may also explain the high variability in the outcomes measured and reported.

Data Collection Methodology

In all, 12 studies used 1 data source and 11 studies used 2 data sources (), which is not different from previous findings []. The results of this study also show that there is high flexibility in data collection methods for MOOC evaluations from survey data to LMS data to more distinct methods such as online focus groups []. The number of participants in the studies was exceedingly varied. This is due to the difference in the data collection methods used. For example, studies with data captured through the LMS, which is capable of capturing data from all of the learners who joined the course, had the highest number of learners. On the contrary, studies that used more time-consuming methods, such as surveys or interviews, generally had a lower number of participants. It is important to note that the MOOC evaluation is not necessarily improved by increasing the number of data sources but rather by conducting a meaningful analysis of the available data. Some studies preferred multiple methods of evaluation and assessment of learning. One paper argued that this allows to evaluate learning of the diverse MOOC population in a more effective way []. Studies should use the best data collection methods to answer their research aims and questions.

Analysis and Interpretation

In total, 16 of 33 studies used only quantitative methods for analysis (), which is in line with the general MOOC research, which has been predominated by quantitative methods [,]. Studies used statistical methods such as descriptive and inferential statistics for data analysis and interpretation of results. The availability of data from sources such as the LMS may have encouraged the use of descriptive statistical methods []. However, 17 of the 33 included studies used some form of qualitative data analysis methods either by using a qualitative study design or by using a mixed methods study design (). This may be explained by the recent (2016-2017) rise in the use of qualitative methods in MOOC research [].

Although inferential statistics can help create better outcomes from studies, this is not always possible. For example, one study [] mentioned a high variation between pre- and postcourse survey participant numbers and another [] mentioned a small sample size as reasons for not using inferential statistical methods. It should be noted that using data from multiple sources and having a large sample size does not guarantee the quality of the evaluation methods.

In MOOC research, qualitative data can be useful to understand the meaning of different behaviors as quantitative data, oftentimes, cannot answer why things happened [].

Outcome Measures

The outcome measures reported greatly varied among studies, which is expected, as identifying the right outcome measures is an inherent challenge in educational research, including more traditional classroom-based studies [].

The choice of evaluation methods is highly dependent on the aim of the evaluation and the size of the MOOCs. For quantitative measures, such as completion and participation rates, metrics can be easily collected through the MOOC platform. However, these metrics alone may be insufficient to provide insights into why students fail to complete the course for future improvement. Although it may be difficult to represent the problem holistically using qualitative methods, it can be useful in providing insights from individuals who participated in the MOOCs. Mixed methods studies combine the 2 modalities to better understand metrics generated and produce greater insights for future improvement of the MOOCs.

Learning outcomes were mostly analyzed by inferential statistical methods owing to the use of pretest and posttest methods and the calculation of gains in learning. This method may be most suited for MOOCs that require knowledge retention. Learning parameters also involved a lot of comparisons, either a comparison with pre-MOOC measures or a comparison with other learners or both. Social interactions were studied in 2 of the MOOC evaluations using social network analysis methods. Although the MOOC completion rate has been often cited as a parameter for MOOC success, it can be noticed that studies started to move away from only using completion rates. For example, studies looked at completion of different steps of the MOOCs or looked at overall completion. The learning outcomes reported in this review should be used with caution as not all of them have been validated or assessed for their reliability except for a few.

Methodological Quality

In total, 26 studies with a cross-sectional design had a low-quality assessment, whereas RCTs and quasi-experimental studies received a high-quality assessment. Having a high level of bias affects the generalizability of studies, which is a common problem in most research using data from MOOCs [,]. The availability of high risks of bias in current MOOC evaluations requires a closer look at what were the sources of bias and what methods can be used to reduce them. The use of not validated, self-reported data sources and the lack of longitudinal data also increases the risk of bias in these studies []. However, although most MOOCs struggle with learner retention and MOOC completion rates [], it is understandable that studies are not able to collect longitudinal data.

Future Directions

The scarcity of studies focusing on the evaluation of the effectiveness of particular MOOCs relative to the number of available studies on MOOCs raises some questions. For example, many studies that were excluded from this review studied MOOC learners or aspects of the MOOCs without conducting an evaluation of course success or effectiveness. As shown in this review, there is a diverse range of evaluation methods, and the quality of these evaluation studies can be as diverse. The motivation of the evaluation exercise should be the basis of the evaluation study design to design effective quantitative or qualitative data collection strategies. The development of general guidance, standardized performance indicators, and an evaluation framework using a design thinking approach can allow these MOOC evaluation exercises to yield data of better quality and precision and allow improved evaluation outcomes. To provide a comprehensive evaluation of MOOCs, studies should try to use a framework to be able to systematically review all of the aspects of the course.

In general, the adoption of a mixed methods analysis considering both quantitative and qualitative data can be more useful for evaluating the overall quality of MOOCs. Although it is useful to have quantitative data such as learner participation and dropout rates, qualitative data gathered through interviews and opinion mining provide valuable insights into the reasons behind the success or failure of a MOOC. Studies of MOOC evaluations should aim to use data collection and analysis methods that can minimize the risk of bias and provide objective results. Whenever possible, studies should use comparison methods, such as the use of pretest or posttest or a comparison with other types of learners, as a control measure. In addition, learner persistence is an important indicator for MOOC evaluation that needs to be addressed in future research.

Strengths and Limitations

To our knowledge, this is the first study to systematically review the evaluation methods of MOOCs. The findings of this review can serve future MOOC evaluators with recommendations on their evaluation methods to facilitate better study designs and maximize the impact of these Web-based platforms. However, as a lot of MOOCs are not necessarily provided by universities and systematically evaluated and published, the scope of this review can only reflect a small part of MOOC evaluation studies.

Conclusions

There is no one way of completing a MOOC evaluation, but there are considerations that should be taken into account in every evaluation. First, because MOOCs are very large, there is a tendency to use quantitative methods using aggregate-level data. However, aggregate-level data do not always tell why things are happening. Qualitative data could further help interpret the results by exploring why things are happening. Evaluations lacked longitudinal data and very few accounted for confounding variables owing to data collection challenges associated with MOOCs such as not having longitudinal data or not having enough data sources. Future studies could help identify how these challenges could be overcome or minimized.

LMS may not report useful findings on an individual level, but they should still be considered and used in MOOC evaluations. Big data in the form of learning analytics can help with decision making, predicting learner behavior, and providing a more comprehensive picture of the phenomena studied []. Studies should still consider using LMS as it can provide a valuable addition to the research, but researchers need to be careful about the depth of the findings that can be concluded from LMS-only datasets.

The use of qualitative data could help enhance the findings from the studies by explaining the phenomena. Both quantitative and qualitative methods could play a key role in MOOC evaluations.

Current MOOC evaluations are subject to many sources of bias owing to the nature of the courses being open and available to a very large and diverse number of participants. However, methods are available to reduce the sources of bias. Studies could use a comparator, such as pretest scores, or other types of learners to be able to calculate relative changes in learning. In addition, studies could control for confounding variables to reduce bias.

This review has provided an in-depth view of how MOOCs can be evaluated and explored the methodological approaches used. Exploring MOOC methodological approaches has been stated as an area for future research []. The review also provided recommendations for future MOOC evaluations and for future research in this area to help improve the quality and reliability of the studies. MOOC evaluations could contribute to the development and improvement of these courses.

# data[].attributes.attachment_link.title
Massive Open Online Course Evaluation Methods: Systematic Review

# data[].attributes.attachment_link.description
Review 1 Global Digital Health Unit, Imperial College London, London, United Kingdom 2 Digitally Enabled PrevenTative Health Research Group, Department of Paediatrics, University of Oxford, Oxford, United Kingdom 3 Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Stockholm, Sweden Corresponding Author Edward Meinert, MA, MSc, MBA, MPA, PhD Digitally Enabled PrevenTative Health Research Group Department of Paediatrics University of Oxford John Radcliffe Hospital Children's Hospital Oxford, OX3 9DU United Kingdom Phone: 44 7824446808 Email: e.meinert14@imperial.ac.uk Background: Massive open online courses (MOOCs) have the potential to make a broader educational impact because many learners undertake these courses.

# data[].attributes.attachment_link.readable_text
How do we know something exists? There are a numbers of ways of knowing...

-Sensory Experience

-Agreement with others

-Expert Opinion

-Logic

-Scientific Method (we're using this one)

The Scientific Process (replicable)

1. Identify a problem

2. Clarify the problem

3. Determine what data would help solve the problem

4. Organize the data

5. Interpret the results

General Types of Educational Research

Descriptive - survey, historical, content analysis, qualitative (ethnographic, narrative, phenomenological, grounded theory, and case study)

Associational - correlational, causal-comparative

Intervention - experimental, quasi-experimental, action research (sort of)

Researchers Sometimes Have a Category Called Group Comparison

Ex Post Facto (Causal-Comparative): GROUPS ARE ALREADY FORMED

Experimental: RANDOM ASSIGNMENT OF INDIVIDUALS

Quasi-Experimental: RANDOM ASSIGNMENT OF GROUPS (oversimplified, but fine for now)

General Format of a Research Publication

Background of the Problem (ending with a problem statement) - Why is this important to study? What is the problem being investigated?

Review of Literature - What do we already know about this problem or situation?

Methodology (participants, instruments, procedures) - How was the study conducted? Who were the participants? What data were collected and how?

Analysis - What are the results? What did the data indicate?

Results - What are the implications of these results? How do they agree or disagree with previous research? What do we still need to learn? What are the limitations of this study?

# data[].attributes.attachment_link.title
Types of Research | Educational Research Basics by Del Siegle

# data[].attributes.attachment_link.description
Types of Research How do we know something exists? There are a numbers of ways of knowing... -Sensory Experience -Agreement with others -Expert Opinion -Logic -Scientific Method (we're using this one) The Scientific Process (replicable) Identify a problem Clarify the problem Determine what data wo ...

# data[].attributes.headline
The ultimate guide to questionnaire development

# data[].attributes.subject
The ultimate guide to questionnaire development

# data[].attributes.attachment_link.readable_text
A questionnaire is a research instrument that consists of a set of questions or other types of prompts that aims to collect information from a respondent. A research questionnaire is typically a mix of close-ended questions and open-ended questions .

Open-ended, long-form questions offer the respondent the ability to elaborate on their thoughts. Research questionnaires were developed in 1838 by the Statistical Society of London.

LEARN ABOUT: Candidate Experience Survey

The data collected from a data collection questionnaire can be both qualitative as well as quantitative in nature. A questionnaire may or may not be delivered in the form of a , but a survey always consists of a questionnaire.

LEARN ABOUT: Testimonial Questions

Advantages of a good questionnaire design

With a survey questionnaire, you can gather a lot of data in less time.

There is less chance of any bias(like selection bias) creeping if you have a standard set of questions to be used for your target audience. You can apply logic to questions based on the respondents' answers, but the questionnaire will remain standard for a group of respondents that fall in the same segment.

Surveying online survey software is quick and cost-effective. It offers you a rich set of features to design, distribute, and analyze the response data.

It can be customized to reflect your brand voice. Thus, it can be used to reinforce your brand image.

The responses can be compared with the historical data and understand the shift in respondents' choices and experiences.

Respondents can answer the questionnaire without revealing their identity. Also, many survey software complies with significant data security and privacy regulations.

LEARN ABOUT: Structured Questionnaire
Characteristics of a good questionnaire

Your survey design depends on the type of information you need to collect from respondents. Qualitative questionnaires are used when there is a need to collect exploratory information to help prove or disprove a hypothesis. Quantitative questionnaires are used to validate or test a previously generated hypothesis. However, most questionnaires follow some essential characteristics:

Uniformity: Questionnaires are very useful to collect demographic information, personal opinions, facts, or attitudes from respondents. One of the most significant attributes of a research form is uniform design and standardization. Every respondent sees the same questions. This helps in data collection and statistical analysis of this data. For example, the retail store evaluation questionnaire template contains questions for evaluating retail store experiences. Questions relate to purchase value, range of options for product selections, and quality of merchandise. These questions are uniform for all customers.

Exploratory: It should be exploratory to collect qualitative data. There is no restriction on questions that can be in your questionnaire. For example, you use a data collection questionnaire and send it to the female of the household to understand her spending and saving habits relative to the household income. Open-ended questions give you more insight and allow the respondents to explain their practices. A very structured question list could limit the data collection.
LEARN ABOUT: Best Data Collection Tools

Question Sequence: It typically follows a structured flow of questions to increase the number of responses. This sequence of questions is screening questions, warm-up questions, transition questions, skip questions, challenging questions, and classification questions. For example, our motivation and buying experience questionnaire template covers initial demographic questions and then asks for time spent in sections of the store and the rationale behind purchases.

As we explored before, questionnaires can be either structured or free-flowing. Let's take a closer look at what that entails for your surveys.

Structured Questionnaires: Structured questionnaires collect quantitative data. The questionnaire is planned and designed to gather precise information. It also initiates a formal inquiry, supplements data, checks previously accumulated data, and helps validate any prior hypothesis.

Unstructured Questionnaires: Unstructured questionnaires collect qualitative data. They use a basic structure and some branching questions but nothing that limits the responses of a respondent. The questions are more open-ended to collect specific data from participants.

Types of questions in a questionnaire

You can use multiple question types in a questionnaire. Using various question types can help increase responses to your research questionnaire as they tend to keep participants more engaged. The best customer satisfaction survey templates are the most commonly used for better insights and decision-making.

Some of the widely used types of questions are:

Open-Ended Questions: Open-ended questions help collect qualitative data in a questionnaire where the respondent can answer in a free form with little to no restrictions.

Dichotomous Questions: The dichotomous question is generally a "yes/no" close-ended question. This question is usually used in case of the need for necessary validation. It is the most natural form of a questionnaire.

Multiple-Choice Questions: Multiple-choice questions are a close-ended question type in which a respondent has to select one (single-select multiple-choice question) or many (multi-select multiple choice question) responses from a given list of options. The multiple-choice question consists of an incomplete stem (question), right answer or answers, incorrect answers, close alternatives, and distractors. Of course, not all multiple-choice questions have all of the answer types. For example, you probably won't have the wrong or right answers if you're looking for customer opinion.

Scaling Questions: These questions are based on the principles of the four measurement scales - nominal, ordinal, interval, and ratio. A few of the question types that utilize these scales' fundamental properties are rank order questions, Likert scale questions, semantic differential scale questions, and Stapel scale questions.

LEARN ABOUT: System Usability Scale

Pictorial Questions: This question type is easy to use and encourages respondents to answer. It works similarly to a multiple-choice question. Respondents are asked a question, and the answer choices are images. This helps respondents choose an answer quickly without over-thinking their answers, giving you more accurate data.

Types of Questionnaires

Questionnaires can be administered or distributed in the following forms:

Online Questionnaire: In this type, respondents are sent the questionnaire via email or other online mediums. This method is generally cost-effective and time-efficient. Respondents can also answer at leisure. Without the pressure to respond immediately, responses may be more accurate. The disadvantage, however, is that respondents can easily ignore these questionnaires. Read more about online surveys.

Telephone Questionnaire: A researcher makes a phone call to a respondent to collect responses directly. Responses are quick once you have a respondent on the phone. However, a lot of times, the respondents hesitate to give out much information over the phone. It is also an expensive way of conducting research. You're usually not able to collect as many responses as other types of questionnaires, so your sample may not represent the broader population.

In-House Questionnaire: This type is used by a researcher who visits the respondent's home or workplace. The advantage of this method is that the respondent is in a comfortable and natural environment, and in-depth data can be collected. The disadvantage, though, is that it is expensive and slow to conduct.

Mail Questionnaire: These are starting to be obsolete but are still being used in some market research studies. This method involves a researcher sending a physical data collection questionnaire request to a respondent that can be filled in and sent back. The advantage of this method is that respondents can complete this on their own time to answer truthfully and entirely. The disadvantage is that this method is expensive and time-consuming. There is also a high risk of not collecting enough responses to make actionable insights from the data.

How to design a Questionnaire

Questionnaire design is a multistep process that requires attention to detail at every step.

Researchers are always hoping that the responses received for a survey questionnaire yield useable data. If the questionnaire is too complicated, there is a fair chance that the respondent might get confused and will drop out or answer inaccurately.

LEARN ABOUT: Easy Test Maker

As a survey creator, you may want to pre-test the survey by administering it to a focus group during development. You can try out a few different questionnaire designs to determine which resonates best with your target audience. Pre-testing is a good practice as the survey creator can comprehend the initial stages if there are any changes required in the survey.

Steps Involved in Questionnaire Design

1. Identify the scope of your research:

Think about what your questionnaire is going to include before you start designing the look of it. The clarity of the topic is of utmost importance as this is the primary step in creating the questionnaire. Once you are clear on the purpose of the questionnaire, you can begin the design process.

The words or phrases you use while writing the questionnaire must be easy to understand. If the questions are unclear, the respondents may simply choose any answer and skew the data you collect.

3. Ask only one question at a time:
LEARN ABOUT: Social Communication Questionnaire
At times, a researcher may be tempted to add two similar questions. This might seem like an excellent way to consolidate answers to related issues, but it can confuse your respondents or lead to inaccurate data. If any of your questions contain the word "and," take another look. This question likely has two parts, which can affect the quality of your data.

4. Be flexible with your options:

While designing, the survey creator needs to be flexible in terms of "option choice" for the respondents. Sometimes the respondents may not necessarily want to choose from the answer options provided by the survey creator. An "other" option often helps keep respondents engaged in the survey.

5. The open-ended or closed-ended question is a tough choice:

The survey creator might end up in a situation where they need to make distinct choices between open or close-ended questions. The question type should be carefully chosen as it defines the tone and importance of asking the question in the first place.

If the questionnaire requires the respondents to elaborate on their thoughts, an open-ended qu estion is the best choice. If the surveyor wants a specific response, then close-ended questions should be their primary choice. The key to asking closed-ended questions is to generate data that is easy to analyze and spot trends.

6. It is essential to know your audience:

A researcher should know their target audience. For example, if the target audience speaks mostly Spanish, sending the questionnaire in any other language would lower the response rate and accuracy of data. Something that may seem clear to you may be confusing to your respondents. Use simple language and terminology that your respondents will understand, and avoid technical jargon and industry-specific language that might confuse your respondents.

For efficient market research, researchers need a representative sample collected using one of the many sampling techniques, such as a sample questionnaire. It is imperative to plan and define these target respondents based on the demographics required.

7. Choosing the right tool is essential:

QuestionPro is a simple yet advanced survey software platform that the surveyors can use to create a questionnaire or choose from the already existing 300+ questionnaire templates.

Always save personal questions for last. Sensitive questions may cause respondents to drop off before completing. If these questions are at the end, the respondent has had time to become more comfortable with the interview and are more likely to answer personal or demographic questions.

Differences between a Questionnaire and a Survey

A questionnaire can is a research instrument that consists of a set of questions to collect information from a respondent.

A survey is a research method used for collecting data from a pre-defined group of respondents to gain information and insights on various topics of interest.

Characteristic

Subset of survey

Consists of questionnaire and survey design, logic and data collection

Read more: Difference between a survey and a questionnaire

Questionnaire Examples

The best way to understand how questionnaires work is to see the types of questionnaires available. Some examples of a questionnaire are:

1. Customer Satisfaction Questionnaire: This type of research can be used in any situation where there's an interaction between a customer and an organization. For example, you might send a customer satisfaction survey after someone eats at your restaurant. You can use the study to determine if your staff is offering excellent customer service and a positive overall experience.USE THIS FREE TEMPLATE

2. Product Use Satisfaction: You can use this template to better understand your product's usage trends and similar products. This also allows you to collect customer preferences about the types of products they enjoy or want to see on the market.USE THIS FREE TEMPLATE

3. Company Communications Evaluation: Unlike the other examples, a company communications evaluation looks at external and internal communications. It can be used to check if the policies of the organization are being enforced across the board, both with employees and clients.USE THIS FREE TEMPLATE

The above survey questions are typically easy to use, understand, and execute. Additionally, the standardized answers of a survey questionnaire instead of a person-to-person conversation make it easier to compile useable data.

The most significant limitation of a data collection questionnaire is that respondents need to read all of the questions and respond to them. For example, you send an invitation through email asking respondents to complete the questions on social media. If a target respondent doesn't have the right social media profiles, they can't answer your questions.

Learn More: 350+ Free Survey Examples and Templates

# data[].attributes.attachment_link.title
Questionnaires: Definition, advantages & examples | QuestionPro

# data[].attributes.attachment_link.description
Learn what a questionnaire is and what are the different types, characteristics and features of a good design. Free examples and tips.

# data[].attributes.headline
DEVELOPING INSTRUMENTS FOR DATA COLLECTION

# data[].attributes.subject
DEVELOPING INSTRUMENTS FOR DATA COLLECTION