# data[].attributes.headline
Authentic Assessment Top 'Reflective' Trumps!

# data[].attributes.subject
Authentic Assessment Top 'Reflective' Trumps!

# data[].attributes.attachment_link.readable_text
The pandemic has undoubtedly shone a light on the need for alternative assessment approaches. Within schools it's sad to see authorities ' hoping' to return to exams; I am rather hoping that we never return to a high stakes examination system since it brings stress and burden to so many. The pandemic has provided a jolt to the education system which we can use as a trigger for change; with regard to technology, this change is advocated through the Gravity Assist report, but the same call to action around assessment is being made by many educational developers and practitioners. There is much we can learn and take forward in relation to assessment, if we now step up and seize the opportunity (said gently, with full awareness of colleague well-being!).

Within higher education the case for authentic assessment is well made across decades of literature, when for most of us a pandemic was just a farfetched Hollywood film. Its well documented benefits include: working as a driver for learning, motivating students, developing higher order thinking skills, developing a wide range of skills, keeping assessment interesting and rewarding, providing opportunities for inclusive practice, helping to fight cheating (though the evidence for this is disputed) and it can feed in to real-world challenges when combined with industry, professions or civic groups.

Authentic assessment could be a tenet of a new way forward, post-pandemic.
In various presentations, and after a synthesis of literature, I have come to see authentic assessment as being characterised by realism, cognitive challenge and evaluative judgment with relevance to self (relevant to my own individual aspirations), discipline community, or professional community. It is assessment which is relevant to me, us or them. Authentic assessment is not just training for work - that would be sadly utilitarian. Ashford-Rowe, Herrington and Brown (2014) helpfully deconstruct the idea of authentic assessment as they identify eight key characteristics which help define the approach, these are: Challenge to the student; Performance or product; Transfer of learning (between contexts or combinations); Metacognition (reflection, self-assessment and evaluation); Recognisable as authentic by a stakeholder; Fidelity (of environment and tools in-use); Discussion and feedback; and, Collaboration.

While the incentive to change away from traditional methods of assessment is clear to many (in light of the the limitations of the status quo), and the vision of what is possible is well explained (in the literature), we may need to do more if change is to follow. I observe, through everyday practice, a widespread apprehension when operationalising the principles of authentic assessment. We need to exemplify what is possible, discuss how existing practice may be adjusted to become more authentic, talk about the workload of change, and we need to gain collective confidence to make changes. Making change to assessment can feel daunting, we need to be clearer about what is OK!
Reflective Cards - Let's Cause a Conversation

As a small contribution to helping this work along, I have created Authentic Assessment Top Reflective Trumps (a clear twist on the game I spent a decade playing with my sons - though based around Star Wars, Cricket and Harry Potter rather than assessment!).

These are a set of cards that can be downloaded. Each card contains an outline of an assessment that may be considered to be authentic. You may not agree, and that's fine 🙂 The assessment type is briefly explained. On each card there also is a panel which lists each of Ashford-Rowe, Herrington and Brown's eight characteristics of authentic assessment. The assessment type on the card is then given a star rating (out of five) for each characteristic. The idea of the cards is that they provide a simple starter for discussions about how assessment may become more authentic, and what options could be drawn upon.

The cards have been formed from my ongoing list of authentic assessments which I have collated over many years - the 'Talking Heads' example was something we used back in 2004/5, so not all of these are new. I have added three specific assessments with attribution, these are great ideas that my colleagues have been working on in recent years. I arrived at the ratings with a great deal of pondering. The ratings, on the extent to which the assessment types provides scope for each characteristic of authentic assessment, are really a personal judgment based on how I imagine the activity to happen; these are not fixed and different people in different contexts would inevitably have different views. The ratings on the card are to work as a stimulus. And before using these with colleagues, you can adjust them as needed.

How to make the most of the cards:
* Flick through the cards and look for ideas of what types of assessment you might try.
* Work with colleagues to shortlist assessments that appeal and discuss their feasibility.
* Discuss the criteria ratings provided for some or all of the assessments and consider whether you agree, and perhaps more importantly, consider how you might strengthen some of the 'scores' by building in specific feedback or collaboration opportunities, or by increasing the level of challenge or by increasing the fidelity of the task. By example in response to one rating a colleague may say something like: "The cards gave an infographic 4 for feedback, but I use this type of assessment and would give it 5 - because when I do it I ensure that there is an opportunity for feedback on the initial ideas and on the outline, I then run a peer poster fair to facilitate peer feedback before submission" - these are the kinds of conversations and peer sharing that I hope that the cards may trigger.
* You might add criteria and ratings that matter to you - so if your institution or school is focussed on prioritising digital skills or creativity, then you may add these as additional lines to the ratings panel and then add your own scores.
* You can fill in the blank cards at the end of the pack as you discuss 'what else?' you may do. If you do add new ideas and want me to add them to the pack (with attribution) then please just mail them to larnold@harper-adams.ac.uk or catch me on Twitter @LydiaJArnold .
* You might even set about making an institutional set of your own cards!

The cards are provided as a PDF and a PowerPoint.

The cards are available under creative commons, so please use and adapt non-commercially, but please refer back to the original source material.

In making the cards, I became even clearer that it is possible to make small changes as well as big ones that move us towards a more authentic assessment diet. We don't all need to throw-out all existing practices to get five stars against each of the characteristics of authentic assessment, we may be able to make small changes to make what we do more authentic and engaging.

There is a world of possibility with authentic assessment, and it is important that the whole student journey hangs together so that feedback from one part of a course feeds in to the next. In moving towards authentic assessment, take care not to introduce so much diversity and demand that students are confused and over-whelmed. A programme level coordination is ideal.

There is a need to evaluate changes that are implemented in your specific context. These cards are not a prescription, they are a reflective tool to cause conversations. To complete the conversation over what works, then be sure to assess the success of any changes (and please go beyond voluntary feedback from the students who say they like it!).

Ashford-Rowe, K., Herrington, J. and Brown, C. (2014) 'Establishing the critical elements that determine authentic assessment', Assessment and Evaluation in Higher Education, 39(2), pp. 205-222. doi: 10.1080/02602938.2013.819566

# data[].attributes.attachment_link.title
Authentic Assessment Top 'Reflective' Trumps!

# data[].attributes.attachment_link.description
As a way of supporting dialogue around authentic assessment, I am offering 'Authentic Assessment Top Reflective Trumps'! These are a set of cards that can be downloaded. Each card contains an outline of an assessment a panel which lists each of Ashford-Rowe, Herrington and Brown's eight characteristics of authentic assessment.

# data[].attributes.headline
Assessment ideas for an AI enabled world

# data[].attributes.subject
Assessment ideas for an AI enabled world

# data[].attributes.body
enabled world

# data[].attributes.headline
Assessment Menu: Designing assessment in an AI enabled world - National centre for AI

# data[].attributes.subject
Assessment Menu: Designing assessment in an AI enabled world - National centre for AI

# data[].attributes.attachment_link.readable_text
The release of ChatGPT coincided with an environment of rising anxiety around academic integrity brought about by the move to online assessment post COVID. This, at least, was the experience of my colleagues and I in the Digital Assessment Advisory at UCL and seemed to reflect concerns elsewhere.

The result was, and remains, a demand for increased assessment security measures such as in-person invigilated exams. This response can be understood not so much as a resistance to change but that the rapid pace of AI developments requires an agile and flexible response, something that HE regulatory systems often cannot accommodate. Add to that the workload involved in changing assessment and negotiations required both internally and, in many cases externally with Professional Statutory and Regulatory Bodies, it is not surprising that familiarity wins out.

Of course, there is a lot at stake; we have a responsibility to protect the integrity of our degrees and ensure that students achieve awards fairly and on their own merit. The question is whether the fall-back position of traditional security-driven assessment can be any more than a stop gap. As Rose Luckin, an expert in both AI and Education, proposes, GenAI is game changing technology for assessment and education in general.

"[GenAI is] much more than a tool for cheating, and retrograde steps (like going back to stressful in-person exams assessing things that AI can often do for us) not only fail to prepare students to engage with the technology, they miss opportunities to focus on what they really need. "

So, what do students really need from assessment in an AI enabled world? In the view of Luckin (and others) AI will change how and what we teach and learn. The types of capabilities students will need to develop and be able to demonstrate include things like critical thinking, contextual understanding, emotional intelligence and metacognition.

Acknowledging the fact that we can't ignore GenAI, ban it or outrun it with detection tools (ethically at least), UCL from the outset adopted the only viable option which was to integrate and adapt to AI in an informed, and responsible way. However, taking this position meant supporting staff navigate this ever-changing terrain.

An AI and Education experts' group was set up in January focussing on four strands: Policy and Ethics, Academic Skills, Assessment design and Opportunities. An output of the group is a newly released Generative AI hub for UCL staff bringing together all the latest information, resources and guidance on using Artificial Intelligence in education, including a three tiered categorisation of AI use in an assessment for staff to guide students on expectations.

As part of the assessment design strand of this group, my contribution was a set of resources called ' Designing assessment in an AI enabled world' divided into two parts:

Small-scale adaptations/enhancements to current assessment practice and which are viable within the existing regulatory framework.

Planning for larger changes with an assessment menu of suggestions for either integrating AI or for which AI would find it difficult to generate a response.

The inspiration for the Assessment menu was Lydia Arnold's ' Top Trumps' , a wonderful resource consisting of 50 suggestions for diversifying assessment using a star rating system based on Ashford-Rowe, Herrington and Brown's characteristics of authentic assessment. Lydia generously made this free to edit and adapt for non-commercial purposes.

Although there was a lot of guidance around using AI in education, it was surprisingly hard to find concrete examples of assessment design for an AI enabled world. I thought Top Trumps would be an ideal starting point and set about trying to reconfigure them to either integrate AI or make it more difficult to generate a response using AI. I also drew on material such as the excellent crowdsourced 'Creative ideas to use AI in education' resource, Ryan Watkin's 'Update your Course Syllabus post', Lydia Arnold's recent integrating AI into assessment practices post and added a few myself. A list of contributors can be found here - Assessment Menu Sources. Examples were tweaked accordingly, framed as student activities and broken down into steps.

I wanted to include meaningful indices for UCL staff to help them identify relevant approaches:

Revised version of the star rating system used in Top Trumps.

Types of assessment corresponding to in UCL's Assessment Framework for taught programmes.

Types of learning that it assesses and develops which references Bloom's taxonomy, but also the types of capabilities experts envisage as being increasingly important.

Appropriate formats.

Where appropriate, whether the activity is suitable for specific disciplinary areas.

My colleague Lene-Marie Kjems helped with visual design and accessibility. We were about to publish our first version when my manager, Marieke Guy, mentioned that a collaborative assessment group at Jisc were also planning an AI version of Lydia Arnold's Top Trumps. There is no such thing as a new idea as Mark Twain famously said!

The Jisc group - Sue Attewell (Jisc), Pam Birtill (University of Leeds), Eddie Cowling (University of York), Cathy Minett-Smith (UWE), Stephen Webb (University of Portsmouth), Michael Webb (Jisc) and Marieke (UCL) - invited me in to work together on producing an interactive, searchable version of the assessment menu for their website. Having forty options to choose from could be overwhelming so being able to filter according to requirements would make it much more usable. We also invited Lydia Arnold (Harper Adams University) to be a consultant and together we worked to review and refine the cards.

Deciding on the search categories was central to the usefulness of the resource. After some discussion we opted to use the categories of summative assessment included in UCL's Assessment Framework as the entry point as these are common in UK Higher Education:

Controlled conditioned exams.

Take home papers/open book exams.

Quizzes and In-class tests.

Practical Exams.

Dissertation.

Coursework and other Written Assessments.

Margaret Bearman and others from Deakin in their highly informative CRADLE webinar series on GenAI suggest we have roughly a year to turn things around in education. In this light, it may seem strange to include Controlled conditioned exams but, however much we want (or need) radical transformation, we felt it would be more helpful to meet people where they currently are, which, as mentioned above, often includes the familiar territory of exam-based assessment.

The Jisc design team did a fabulous job of turning the original PPT version into the interactive resource you can now find on the Publications and resources page (scroll down to locate it). Using clickable icons representing each of the above (on slide 7), users can be presented with options to consider.
Image: example search results for Take-Home Papers/Open Book
Like any AI related guidance we produce right now, this will need to be updated as the technology and assessment practice evolve together.

The most frequent questions I am currently asked are 'How can I make my assessments AI proof?' and 'How can I stop students using AI when they're not supposed to?' This resource does not promise to solve either of these questions, but the hope is that this genuinely collaborative menu of assessment ideas, drawing on expertise from UK and beyond, will provide inspiration for educators and support them to work alongside students to engage productively and thoughtfully with AI in their assessments.

# data[].attributes.attachment_link.title
Assessment Menu: Designing assessment in an AI enabled world - National centre for AI

# data[].attributes.attachment_link.description
The release of ChatGPT coincided with an environment of rising anxiety around academic integrity brought about by the move to online assessment post COVID. This, at least, was the experience of my colleagues and I in the Digital Assessment Advisory at UCL and seemed to reflect concerns elsewhere.

# data[].attributes.headline
Tailoring university assessment in the age of ChatGPT

# data[].attributes.subject
Tailoring university assessment in the age of ChatGPT

# data[].attributes.attachment_link.readable_text
OpenAI has just released ChatGPT-4, the latest iteration of its powerful AI text generator. The tool is capable of generating convincingly human-like responses to almost any question users put to it.

It can write limericks, tell jokes, and plot a novel. And it can draft a convincing response to almost any question a high school teacher or university lecturer might ask students to write about.

Previous iterations of OpenAI's GPT language models would often generate text riddled with strange and obvious mistakes. ChatGPT (and particularly GPT-4) is less prone to these problems. The responses that ChatGPT generates are apparently capable of passing exams across many disciplines.

It's tempting to think we'll always be able to distinguish between the work of AI and the work of humans, particularly when it comes to distinctly "human" tasks such as creative writing, careful reasoning, and drawing novel connections between different kinds of information.

Unfortunately, this optimism is misguided. AI-generated prose and poetry can be beautiful. And with some clever prompting, AI tools can generate passable argumentative essays in philosophy and bioethics.

This raises a serious worry for universities that students will be able to pass assessments without writing a single word themselves - or necessarily understanding the material they're supposed to be tested on.

This isn't just a worry about the future; students have already begun submitting AI-generated work.

Cheating bans hard to enforce

Some institutions treat the use of AI text-generators as "cheating". For example, public school systems in NSW and Victoria have banned ChatGPT.

But such bans will be hard to enforce. Compared to traditional forms of plagiarism (where plagiarised work can be compared directly with the source material), student use of AI-generated text is hard to detect - and harder still to prove, in part because new ChatGPT generates new responses each time a user inputs the same prompt.

One might hope the problem is just temporary. For its part, OpenAI (alongside other developers) is developing tools to detect AI-assisted "cheating" - though such tools are prone to making mistakes, and can at present be circumvented by asking ChatGPT to write in a style that its own detector is unlikely to catch.

OpenAI is also developing a technique for " watermarking " AI-generated text. This works by influencing which words the AI chooses, leaving subtle quirks in the writing that can be detected further down the track.

But this strategy also has serious limitations. Lightly editing AI-generated text would presumably overwrite these watermarks. And students could seek out alternative AI tools that don't watermark AI text.

The educational challenges of AI text generation are here to stay.

Adapting won't be an easy process

Generative AI tools such as ChatGPT are poised to make far-reaching changes to how we approach writing tasks.

These tools are exciting. Among other things, they'll make some tedious and difficult parts of the writing process easier (including for students for whom English is a second language).

But adapting to a world where AI can write compelling student essays will not be straightforward.

Sam Altman, the CEO of OpenAI, has compared the release of ChatGPT to the advent of the calculator. Calculators brought about enormous benefits; ChatGPT will, Altman claims, do the same.

Schools have adapted to calculators by changing how maths is tested and taught; we now need to do the same for ChatGPT.

This analogy seems more or less right, but its implications are less reassuring than Altman intends. The task of adapting to calculators has been long and arduous, and it remains unclear how best to incorporate them into the classroom.

Mathematicians continue to raise plausible worries that overuse of calculators impedes the development of important forms of " mathematical thinking ".

Rather than comforting us, the parallel with calculators should alert us to the magnitude of the task we face.

The two main threats

We see two main threats posed by tools like ChatGPT.

The first is that they'll produce content that's superficially plausible but entirely incorrect.

AI outputs can thus leave us with a deeply mistaken picture of the world.

Contrary to appearances, ChatGPT is not "trying" (but, often, failing) to assert facts about the world. Instead, it is (successfully) performing a different task - that of generating superficially plausible or convincing responses to a prompt.

In humans, we would describe this kind of behaviour as "bullshitting". The term also aptly describes many AI outputs.
Read more: ChatGPT: We need more debate about generative AI's healthcare implications
The second worry is that reliance on these tools will result in the erosion of important skills. Essay writing, for example, is valuable in part because the act of writing can help us think through difficult concepts and generate new ideas.

Good arguments often emerge in an iterative fashion over several drafts, where the final version looks quite different to what one envisaged in their original plan.

Writing, so understood, is not just an act of translating our thoughts onto the page; for many writers, it's part of the reasoning process. And if we begin to outsource the writing process to artificial intelligence, it's a skill students might lose if they come to rely on AI text generators.

Addressing the challenges

How, then, should we meet these challenges? In these early stages of the introduction of generative AI, educators may feel overwhelmed by the rapidly changing technological environment, but students are also coming along for the ride with us.

Being transparent about decisions we make regarding when and how AI tools are used in assessment could help ease anxiety on both sides.

We suggest four approaches.

Teach students how (not) to use AI text generators

ChatGPT can be a useful tool. It can, for instance, help generate ideas and get words on the page.

The worries about misinformation are serious. But these are best addressed by teaching students how to use these tools, how to understand their limitations, and how to fact-check their output.

Fortunately, the core skills cultivated by a good education provide a strong foundation for this project. Teaching students how to read critically, how to evaluate or corroborate evidence, and how to distinguish good arguments from bad, are things universities should be doing already.

One approach might be to develop specific assessment tasks where students generate, analyse, and criticise AI outputs.

While such tasks might have some role to play, we would caution against placing generative AI at the centre of education. There's no shortage of bad arguments being put forward by human beings. We should teach students to recognise bullshit wherever it appears.

Continue efforts to design 'authentic' assessments

We should remind ourselves that for most, choosing to participate in higher education comes from a genuine interest in a subject.

This fact may go some way towards mitigating the temptation to outsource their studies to AI, particularly when the value of completing this work is clear to students.

By designing assessments that are relevant to students' future careers, and clarifying the purpose of tasks in relation to their own development, we can encourage learners to engage with assessment in the way we intended.

Assessment that engages with, and leverages, students' interests could motivate learners to remain engaged such that they don't see value in outsourcing the pursuit of their own knowledge to AI.

This is a desirable way of designing assessment, both as a response to ChatGPT and as a general goal for higher education.

But while this strategy may undercut some students' desire to turn to AI, it's unlikely to fully dissolve the temptation to save time by having an AI write your essay - particularly where doing so would not lower (and might even raise) your final grade.

Additional strategies are needed.

Balance essays with other types of assessment

A key worry about AI text-generation is that students won't actually understand what they appear to, given the work they've submitted.

This concern can be met by balancing written work with other kinds of assessments.

In particular, in-person oral presentations cannot be taken over by any algorithm, and so may be an ideal option (provided, of course, that any increase in workload for teaching staff is supported by the institution).

Supplementing traditional essays with other assessments need not come at the expense of good assessment design.

To the contrary, there are good educational reasons to vary written work with these other kinds of assessment; oral communication skills are enormously valuable across a range of professions.

At the same time, writing skills are poised to become less important as ChatGPT and tools like it start to play a major role in how text is produced, both within and outside the classroom.

Develop AI-resistant written assessments

Another strategy involves designing assignments where students are either required to demonstrate their own understanding (independently of written work they've submitted), or where AI-generated bullshit can be easily detected.

The revival of pen-and-paper exams is being considered by Australian universities.

This strategy may have a role to play, but it would come at a cost. We're amid a shift away from pen-and-paper examinations to "authentic" assessments - that is, assessments that evaluate skills students will employ in real-world settings.

Few workplaces require their employees to write detailed discussions of difficult questions by hand, in isolation, and without the ubiquitous modern conveniences of an internet connection and a word processor.
Read more: Rising from the ashes: Higher education in the age of AI
An alternative is to combine written essays with the presentation and discussion of this work during class time, potentially modelled after the format of a viva presentation or thesis defence (albeit made gentler and shorter according to the cohort being taught.)

The viva format requires students to understand, and be able to communicate, the ideas they've defended in their essay - regardless of whether they outsourced any of the writing to an algorithm.

Finally, there's some scope to develop essay questions that ChatGPT is unlikely to perform well on.

In our own experiments, we found ChatGPT can generate convincing responses about major works in our respective disciplines.

However, it fares very poorly when asked about the cutting edge of scholarly debate, since the corpus of work it was trained on contains much less (if any) discussion of this work.

When asked to reference its claims, it's prone to "hallucinate" sources that don't exist.

Assignments that ask for deep research on recent scholarly developments are, for now, relatively AI-resistant. But this strategy will need to be continuously reviewed as AI language models continue to improve.

Picking the right tool for the job

Assessment regimes should be tied to the educational goals we're trying to achieve, not a knee-jerk reaction to new technologies.

We recommend that educators considering how to adapt their assessment regime in light of ChatGPT ask themselves the following questions:

These suggestions differ from some existing responses, which centre on detecting or preventing the use of AI tools.

This is for a good reason. Our students will enter a world where the use of such tools will likely be commonplace. Indeed, ChatGPT has already been listed as an author on recent academic papers, and utilised by journalists and authors.

We need to prepare students for a world where tools like ChatGPT are readily available - and to teach them how to use these tools well.
Read more: ChatGPT: Old AI problems in a new guise, new problems in disguise
In the longer term, progress in generative AI seems likely to pose even more profound challenges to educators.

In the not-too-distant future, ChatGPT and tools like it may become better than most humans at writing in many genres and on many topics.

It's far from clear whether students will bother to learn to write in such circumstances - or whether employers will care whether they do.

Given the connection between learning to write and learning to think, this is a disheartening prospect.

Moreover, it's almost certain generative AI will become capable of tasks that many educators think of as central to their own profession. ChatGPT can already generate useful feedback on written work.

Non-profit education organisation Khan Academy recently announced it will be launching a GPT-4 powered AI-assistant, Khanmigo, which can play the role of a tutor to students, and can also assist teachers with tasks such as creating lesson plans.

Dystopian visions in which AI teachers set tasks that students then farm out to AI look all too plausible.

The immediate challenge for educators is to determine what an AI-literate skill set looks like, and how to evaluate whether students have these skills, especially when many of us are new to these skills ourselves.

If we can get this right, perhaps we'll be able to continue to teach and assess the core skills of research, reflection, argument, and critical thinking that have traditionally sat at the heart of the university.

The deeper challenge posed by the "threat" of AI is to imagine what education would look like should the tools available to us relieve us of the need to exercise these crucial skills.

# data[].attributes.attachment_link.keywords[]
faculty medicine nursing and health science

# data[].attributes.attachment_link.title
Tailoring university assessment in the age of ChatGPT

# data[].attributes.attachment_link.description
The educational challenges of AI text generation are here to stay. The challenge for educators is to determine what an AI-literate skill set looks like, in order to continue to teach and assess the core skills that have traditionally sat at the heart of the university. Read more at Monash Lens.

# data[].attributes.headline
Chatting and cheating: Ensuring academic integrity in the era

# data[].attributes.subject
Chatting and cheating: Ensuring academic integrity in the era

# data[].attributes.body
of ChatGPT

# data[].attributes.headline
Conceptualizing AI literacy: An exploratory review

# data[].attributes.subject
Conceptualizing AI literacy: An exploratory review

# data[].attributes.headline
Engaging with AI in your education and assessment

# data[].attributes.subject
Engaging with AI in your education and assessment

# data[].attributes.attachment_link.readable_text
Guidance on how you might engage with this in your assessments effectively and ethically.

There is a lot of interest in and experimentation with artificial intelligence (AI) tools such as ChatGPT, DALLE-2, CoPilot, and most recently Google Bard.

All of these can be helpful tools for generating content that might contribute to assessed work, which has led some institutions to prohibit their use.

We believe these tools are potentially transformative as well as disruptive, that they will feature in many academic and professional workplaces, and that rather than seek to prohibit your use of them, we will support you in using them effectively, ethically and transparently.

Whilst they are attractively powerful and easy to use, they can also provide misleading or incorrect information, and can negatively impact your learning as they can offer shortcuts that reduce the need for critical engagement that is key to deep and meaningful learning. You also need to be aware of the difference between reasonable use of such tools, and at what point their use might be regarded as giving you an unfair advantage.

It is important you do not use AI tools to generate an essay and submit it as if it was your own work. In this briefing, we explain how AI might be used and how to acknowledge its use.

What is AI good for?

These tools can help students in a number of ways - for instance

Answering questions where answers are based on material which can be found on the internet.

Drafting ideas and planning or structuring written materials.

Generating ideas for graphics, images and visuals.

Reviewing and critically analysing written materials to assess their validity.

Helping to improve your grammar and writing structure - especially helpful if English is a second language.

Experimenting with different writing styles.

Getting explanations.

Debugging code.

Getting over writer's block.

Limitations

However, artificial and human intelligence are not the same; these AI tools do not understand anything they produce nor do they understand what the words they produce mean when applied to the real world.

Open.ai, the creators of ChatGPT, have provided helpful guidance for educators and students - in short:

Whilst their output can appear plausible and well written, AI tools frequently get things wrong and can't be relied upon for factual accuracy.

They perform better in subjects which are widely written about, and less well in niche or specialist areas.

They cannot currently provide references - they fabricate well formatted but fictitious citations.

They can perpetuate stereotypes, biases and Western perspectives.

More fundamentally, overreliance on these tools will reduce your opportunities to hone your writing, critical thinking, and evaluation skills - key academic and professional skills which you will need as you progress through your studies and beyond. What these tools can provide, however, is an opportunity to build critical analysis and evaluation skills - perhaps by studying and critiquing what they produce, and making judgements about whether what they produce is actually valid and believable.

AI and academic misconduct

When you submit work via Moodle or AssessmentUCL you are reminded that:

Using AI tools to help with such things as idea generation or your planning may be an appropriate use, though your context and the nature of the assessment must be considered. It is not acceptable to use these tools to write your entire essay from start to finish. Also, please bear in mind that words and ideas generated by some AI tools make use of other, human authors' ideas without referencing them, which, as things stand, is controversial in itself and considered by many to be a form a plagiarism.

Your teachers will be able to provide more specific guidance on what tools, if any, are allowed in the context of your own assignments, but the following applies across all of UCL's assessments.

When you hide something and aren't transparent about your approach.

When you take something directly from someone (or something) else and don't reference or acknowledge.

When you don't use something in an appropriate way.

When you don't follow the assessment guidance.

If teaching staff suspect that you are trying to pass off AI-generated output as your own work then the regulations in the  Academic Manual  (9.2.1 g, h and m) apply. You would be likely to be invited to an investigatory viva to probe the authorship of your work

How to acknowledge AI sources in your work

If your teachers have made clear that you may use AI sources in your assessed work, then you should acknowledge, describe and reference its use. The Library Skills pages share guidance on how to acknowledge and reference AI in your work.
Find out how to reference AI

# data[].attributes.attachment_link.title
Engaging with AI in your education and assessment

# data[].attributes.attachment_link.description
Guidance on how you might engage with this in your assessments effectively and ethically.

# data[].attributes.attachment_link.description
A series of instructional web pages for instructors and teaching teams about using AI tools for teaching and learning.

# data[].attributes.headline
What do university students know about Artificial Intelligence? Development and validation of an AI literacy test

# data[].attributes.subject
What do university students know about Artificial Intelligence? Development and validation of an AI literacy test

# data[].attributes.attachment_link.description
About ScienceDirect
Shopping cart
Contact and
supportTerms and
conditionsPrivacy policy

# data[].attributes.attachment_link.description
About ScienceDirect
Shopping cart
Contact and
supportTerms and
conditionsPrivacy policy

# data[].attributes.headline
Thinking about Assessment in the Time of Generative Artificial Intelligence | Digital Futures Institute (DFI) | Teachers College, Columbia University

# data[].attributes.subject
Thinking about Assessment in the Time of Generative Artificial Intelligence | Digital Futures Institute (DFI) | Teachers College, Columbia University

# data[].attributes.attachment_link.title
Thinking about Assessment in the Time of Generative Artificial Intelligence | Digital Futures Institute (DFI) | Teachers College, Columbia University

# data[].attributes.attachment_link.description
Teachers College, Columbia University, is the first and largest graduate school of education in the United States, and also perennially ranked among the nation’s best.

# data[].attributes.attachment_link.readable_text
Academic and technologies teams at Barnard College developed an AI literacy framework to provide a conceptual foundation for AI education and programming eﬀorts in higher education institutional contexts.

Credit: SuPatMaN / Shutterstock.com © 2024

Barnard College is an undergraduate liberal arts college for women. It is both a distinct institution and a member of the broader Columbia University ecosystem in New York City. Several small but agile campus teams are working to advance conversations about generative artificial intelligence (AI) topics. As members of Instructional Media and Academic Technology Services (IMATS) and the Center for Engaged Pedagogy (CEP), we have developed educational programming on various AI topics for the Barnard community. Over the past year, we have held open lab sessions to test diﬀerent text- and image-based AI tools, hosted guest speakers on copyright and fair use, facilitated generative AI syllabus statement workshops for faculty, conducted instructional workshops (GenAI 101), and led individualized education sessions for faculty departments. This process has been ongoing and iterative as the tools shift and campus community members' needs change. We have also put internal survey, evaluation, and feedback mechanisms in place to better understand faculty and staff needs related to the use of generative AI tools.

The Need for AI Literacy

Barnard College has established several internal working groups and task forces to discuss bigger questions about the impact of AI on our institution. Currently, there is no mandate or recommendation for faculty to embrace or ban AI in their classrooms. However, faculty are encouraged to deﬁne and discuss their expectations around the use of AI in their assignments. (The CEP has created many faculty resources, including decision trees to guide faculty planning, sample syllabus statements, assignments integrating generative AI, and other materials, to help guide decision-making about whether and how to incorporate generative AI in the classroom.) Higher levels of AI literacy can help faculty make informed decisions about the use of AI in their courses and assignments. On the academic technology services side, the IMATS team has decided not to implement or pursue AI surveillance technology to monitor academic integrity due to bias and questionable reliability of these tools. However, the landscape and corollary policies could change as generative AI technologies evolve.

A Framework for AI Literacy

Members of the IMATS and CEP developed the following framework to guide the development and expansion of AI literacy among faculty, students, and staff at Barnard College. Our framework provides a structure for learning to use AI, including explanations of key AI concepts and questions to consider when using AI. The four-part pyramid structure was adapted from work done by researchers at the University of Hong Kong and the Hong Kong University of Science and Technology (see figure 1). (The Hong Kong researchers' work builds upon Bloom's Taxonomy.) The framework is intended to meet people where they are and scaffold upon their current AI literacy level, whether they have little to no knowledge of AI or are prepared to build their own large language model (LLM). It breaks AI literacy into the following four levels:

Understand AI

Use and Apply AI

Analyze and Evaluate AI

Create AI

When applying this framework, it is important to keep in mind that AI is a broad field. There are many types of AI, both real and theoretical. While the concepts enumerated in this article focus primarily on generative AI, the overall structure of the framework can be applied to other forms of AI and technology literacies (e.g., cybersecurity). And, while the information in each level builds on the concepts discussed in the previous level, it is not necessary to learn everything at one level before moving on to the next. For instance, when analyzing how generative AI could impact the labor market, understanding how generative AI models are trained is helpful; however, it's not necessary to master the intricacies of neural networks to conduct such an analysis.

Level 1: Understand AI

The bottom of the pyramid covers basic AI terms and concepts. Most of the programming and instruction at Barnard has focused on levels one and two (understanding, using, and applying AI), as this is a rapidly evolving technology, and there is still a lot of unfamiliarity with it.
Core Competencies

Be able to define the terms "artificial intelligence," "machine learning," "large language model," and "neural network"

Recognize the benefits and limitations of AI tools

Identify and explain the differences between various types of AI, as defined by their capabilities and computational mechanisms

Key Concepts

Artificial intelligence, machine learning, artificial neural networks, large language models, and diffusion models

Artificial narrow intelligence, artificial general intelligence, artificial super intelligence, reactive machines, limited memory, theory of mind, and self-aware

AI tools, such as ChatGPT (see figure 2), Siri, Alexa, Deep Blue, and predictive text

Technical frameworks related to AI (open-source models versus closed models, APIs and how they are used)

Reflection Questions

What type of AI is this?

What technologies does this AI tool utilize?

What was this tool designed to do? What kind of information does it accept as input and return as a response (text, video, audio, etc.)?

What might this tool be particularly useful for?

What would it not be useful for?

Level 2: Use and Apply AI

The second level of AI fluency indicates that users can use tools like ChatGPT to achieve their goals; these users are familiar with prompt engineering techniques and know how to collaboratively refine, iterate, and edit with generative AI tools. Programming designed to build level two fluency at Barnard College includes hands-on labs and real-time collaborative prompt engineering (see figure 3).
Core Competencies

Successfully utilize generative AI tools for desired responses

Experiment with prompting techniques and iterate on prompt language to improve AI-generated output

Review AI-generated content with an eye toward potential "hallucinations," incorrect reasoning, and bias

Key Concepts

Prompt engineering, context windows, hallucinations, bias, zero-shot prompting, and few-shot prompting

Prompting techniques for text-based generative AI, such as adding specificity, using context and details, and asking the model to consider pros and cons or evaluate alternative positions

Privacy, conﬁdentiality, and copyright considerations for information fed into prompting tools

Reflection Questions

Why did a prompt generate a particular response?

How could the prompt be adjusted to get a different response?

What strategies can be used to reduce bias and hallucinations?

How can AI output be checked for bias and hallucinations?

Level 3: Analyze and Evaluate AI

Analyzing and evaluating AI involves a more complex meta-understanding of generative AI. At this level, users should be able to critically reflect on outcomes, biases, ethics, and other topics beyond the prompt window. An example of programming at this level is an event that featured an expert who discussed the current copyright and intellectual property questions surrounding AI and the environmental and climate impacts that generative AI might have (see figure 3). Of course, one can participate in conversations about these questions and ideas without knowing all of the AI definitions. However, familiarity with the preceding levels in the pyramid informs one's baseline understanding and vocabulary, helping the individual to understand how AI intersects with other fields.
Core Competencies

Examine AI in a broader context, bringing in knowledge from one's discipline or interests

Critique AI tools and offer arguments in support of or against their creation, use, and application

Analyze ethical considerations in the development and deployment of AI

Key Concepts

Critical perspectives on AI (The following examples are not intended to be all-encompassing.)

environmental sustainability

labor

privacy

copyright

race, gender, class, and other biases

misinformation

Reflection Questions

What other perspectives or frameworks might be useful in assessing the implications of using generative AI tools?

Where might biases in AI come from?

In what ways does the use of generative AI tools align with or diverge from your personal values?

Level 4: Create AI

At this level of AI fluency, users are able to engage with AI at a creator level. For example, users can build on open APIs to create their own LLM or leverage AI to develop new systems (see figure 5). Currently, Barnard offers less programming at level four than at the other three levels, but there have been workshops at the Computational Science Center that provide technical instruction related to building AI and machine-learning models. Engaging people at all levels of AI fluency is important.
Core Competencies

Synthesize learning to conceptualize or create new ideas, technologies, or structures that relate to AI. Reaching this level of literacy could include the following:

Conceive of novel uses for AI

Build software that leverages AI technology

Propose theories about AI

Reflection Questions

What is uniquely human about your ideas, technologies, or structures? How might they differ from what an AI could create?

What specific AI features lend unique affordances to ideas, technologies, or structures?

Conclusion and Next Steps

While this AI literacy framework is not exhaustive, it provides a conceptual foundation for AI education and programming efforts, particularly in higher education institutional contexts. The intention is to maintain neutrality with AI use, recognizing that technology literacy can lead to the decision not to use it. The impact of AI on higher education will likely be significant, impacting admissions, research, and curricula. Education and foundational literacy are the first steps for a community to engage productively with this rapidly changing technology.

There are many possible next steps Barnard College might take related to generative AI, but specific to the AI literacy framework, the IMATS and CEP teams may explore "moving up" the literacy pyramid in programming, resources, and events as awareness and basic literacy grow. Currently, the majority of our offerings are at levels one and two, but we hope to shift our programming focus to levels two and three. A recent survey revealed that a significant number of faculty and students still have never used generative AI and have negative perceptions of these tools, so our teams are also exploring ways to better facilitate practical and critical engagement.

Another goal of the AI literacy initiative is to underline the human aspect of these technologies. While using generative AI can almost feel like alchemy-spinning gold from plain text via black box technology-it is very much built upon human knowledge, which has its own biases and inequities. Using a critical lens when engaging with generative AI may help users identify existing biases and prevent users from exacerbating them.

Notes

Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou, "GPT Detectors Are Biased Against Non-native English Writers," Patterns 4, no. 7 (July 2023): 100779.

Davy Tsz Kit Ng, Jac Ka Lok Leung, Samuel Kai Wah Chu, and Maggie Shen Qiao, "Conceptualizing AI Literacy: An Exploratory Review," Computers and Education: Artificial Intelligence 2 (2021); Benjamin S. Bloom, Max D. Engelhart, Edward J. Furst, Walker H. Hill, and David R. Krathwohl, Taxonomy of Educational Objectives: The Classification of Educational Goals, Handbook I: Cognitive Domain (New York: David McKay, 1956).

Melanie Hibbert is Director of Instructional Media and Academic Technology Services and the Sloate Media Center at Barnard College, Columbia University.

Elana Altman is Senior Associate Director for UX and Academic Technologies at Barnard College, Columbia University.

Tristan Shippen is Senior Academic Technology Specialist, Barnard Library and Academic Information Services at Barnard College, Columbia University.

Melissa Wright is Executive Director, Center for Engaged Pedagogy, at Barnard College, Columbia University.

# data[].attributes.attachment_link.description
Academic and technologies teams at Barnard College developed an AI literacy framework to provide a conceptual foundation for AI education and programm

# data[].attributes.headline
Former
à l’éthique de l’IA
en enseignement
supérieur :
Référentiel
de compétence

# data[].attributes.subject
Former
à l’éthique de l’IA
en enseignement
supérieur :
Référentiel
de compétence

# data[].attributes.headline
A systematic review of AI literacy conceptualization, constructs, and
implementation and assessment efforts (2019–2023)

# data[].attributes.subject
A systematic review of AI literacy conceptualization, constructs, and
implementation and assessment efforts (2019–2023)

# data[].attributes.headline
Developing a holistic AI literacy assessment matrix – Bridging generic, domain-specific, and ethical competencies

# data[].attributes.subject
Developing a holistic AI literacy assessment matrix – Bridging generic, domain-specific, and ethical competencies

# data[].attributes.headline
AI Literacy Competency Framework for Educators & L&D Professionals*

# data[].attributes.subject
AI Literacy Competency Framework for Educators & L&D Professionals*

# data[].attributes.headline
Teachers’ AI digital competencies and twenty‑first century

# data[].attributes.subject
Teachers’ AI digital competencies and twenty‑first century

# data[].attributes.body
skills in the post‑pandemic world

# data[].attributes.headline
Can AI Provide Useful Holistic Essay Scoring?

# data[].attributes.subject
Can AI Provide Useful Holistic Essay Scoring?

# data[].attributes.attachment_link.readable_text
Abstract

Researchers have sought for decades to automate holistic essay scoring. Over the years, these programs have improved significantly. However, accuracy requires significant amounts of training on human-scored texts-reducing the expediency and usefulness of such programs for routine uses by teachers across the nation on non-standardized prompts. This study analyzes the output of multiple versions of ChatGPT scoring of secondary student essays from three extant corpora and compares it to quality human ratings. We find that the current iteration of ChatGPT scoring is not statistically significantly different from human scoring, but exact agreement with humans is still difficult. Consistency and agreement within one point, however, is achievable and may be sufficient for low-stakes, formative assessment purposes.

License

CC-By Attribution 4.0 International

# data[].attributes.attachment_link.title
Can AI Provide Useful Holistic Essay Scoring?

# data[].attributes.attachment_link.description
Researchers have sought for decades to automate holistic essay scoring. Over the years, these programs have improved significantly. However, accuracy requires significant amounts of training on human-scored texts-reducing the expediency and usefulness of such programs for routine uses by teachers across the nation on non-standardized prompts.

# data[].attributes.attachment_link.readable_text
As a former secondary English teacher, senior examination assessor, and lecturer for initial teacher education, I understand the allure of using Generative AI (GenAI) for grading student work. We're all familiar with the workload of assessment and reporting. The idea of a tool that could save time and streamline the grading process is undeniably appealing. It's no surprise, then, that the market is flooded with AI-powered grading solutions, all promising to make our lives easier.

However, as I've explored in previous posts on the capabilities and limitations of GenAI, I firmly believe that this technology is fundamentally unsuited for high-stakes student assessment. At its core, GenAI generates probabilistic outputs based on patterns in training data, lacking true understanding and the ability to make qualitative judgments. This leads to inconsistency and bias in grading, raising serious concerns about fairness and reliability.

The use of AI in grading also raises a host of ethical and equity issues. As I wrote in " Generative AI doesn't 'democratize creativity' ", the notion that AI levels the playing field is often an illusion. In reality, relying on AI for grading may exacerbate existing inequities and privilege certain groups of students over others.

In this post, I'll go deeper into the reasons why I believe GenAI should not be used for grading, drawing on recent experiments and real-world examples. I'll also explore the potential risks and unintended consequences of AI-powered assessment. By the end, I hope to convince you that, despite the temptation, GenAI is a dead-end when it comes to evaluating student work.

The fundamental limitations of LLMs

At the heart of the problem with using GenAI for grading is the fundamental way these systems work. GenAI produces outputs based on probabilistic patterns in its training data, without any real understanding, reasoning, or ability to make qualitative judgments. Essentially, it's making what appear to be educated guesses, but are in effect just statistical patterns.

This means that the grades GenAI assigns can vary significantly based on seemingly minor differences in prompt language or details in the student work. In a quick experiment, I fed the same Year 9 persuasive writing piece into ChatGPT multiple times, changing only the student name. The grades ranged from 78 to 95 out of 100 - a massive discrepancy based on a single variable.

Such inconsistency completely undermines the fairness and reliability of the grading process. It creates a false sense of objectivity and accuracy, when in reality, the grades are no more meaningful than a roll of the dice.

But it gets worse. GenAI models are trained on vast datasets scraped from the internet, which can encode all sorts of societal biases and discrimination. The models can make inferences about student attributes like race, gender, or background based on their writing, potentially disadvantaging certain groups.

And here's the rub: this kind of bias is a lot harder to detect and address compared to human grader bias. We've got strategies like anonymisation and moderation to mitigate human bias, imperfect as they are. But with AI, simply removing the student's name doesn't cut it. The bias is baked into the model at a much deeper level, based on the training data and the patterns it's picked up.

So not only are the grades inconsistent and unreliable, they're also likely to be biased in ways we can't easily control for. It's a recipe for disaster when it comes to fair, equitable assessment.

Improving the prompt only improves the appearance of accuracy

One of the biggest criticisms in the (lengthy) comment thread alongside the original post was that my prompt was overly simplistic. Why didn't I use a more elaborate prompt, guiding the LLMs response with criteria, an assignment sheet, or maybe even exemplar marked student work?

It's a fair question. And yes, more detailed prompts can help anchor the AI's responses and make them somewhat more consistent. But let's be clear - it's a superficial fix. The AI still doesn't have any deep understanding of the work it's grading. It's just getting better at pattern-matching and churning out responses that fit the rubric.

Others argued that human graders are biased and inconsistent too. And that's of course true to an extent. But here's the thing: we know about human bias, and we've got strategies to mitigate it, like blind marking and moderation. We invest in professional development for markers. When I assessed the senior English certificate examination (VCE English), we marked blind, double marked, and had a qualified team of expert assessors with years of training to moderate results. And crucially, human graders have the capacity for contextual understanding and empathy that AI just doesn't.

The argument that "AI is biased but so are humans," is a false equivalence: apples and oranges. While it's true that both AI and humans can exhibit biases, the nature and impact of those biases are not comparable. Human biases stem from individual experiences, cultural backgrounds, and societal influences, whereas AI biases are typically the result of biases present in the training data or introduced by the designers. Additionally, human educators can actively work to recognise and mitigate their biases, while AI systems lack this self-awareness and agency.

Bias is an incredibly complex problem in AI systems, and there might be even more subtle issues than we're aware of. It seems as though the more sophisticated models become, the better they are at inferring details about authors based on even minor details in texts. For example, the output of an AI response to an assessment item can be influenced dramatically by changing only one word, as in these experiments by Melissa Warr and Punya Mishra where they changed one word ("classical" for "rap" music) and it impacted the outcome.

These findings are supported in the article Dialect prejudice predicts AI decisions about people's character, employability, and criminality by Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, Sharese King who demonstrate that "dialect prejudice has the potential for harmful consequences by asking language models to make hypothetical decisions about people, based only on how they speak." This could, of course, impact students through their writing if AI is used to grade written assessments.

I often hear the argument that AI is improving at a breakneck pace, and these issues will be ironed out "any day now". It's the "this is the worst AI we'll ever use" argument favoured by people like Professor Ethan Mollick. But while the tech is certainly developing fast, the fundamental problems aren't going away.

Bigger language models and more training data alone won't magically give AI genuine understanding or eliminate bias. We can't take the hype and marketing at face value. As educators, we need to critically evaluate the evidence and implications, not just jump on the bandwagon.

The other equity issues of GenAI grading

One of my biggest concerns about using AI for grading is the potential for it to worsen existing equity gaps in education: access to AI tools is far from equal.

Imagine a student from a low-income family, living in a remote area with patchy internet. They're relying on free, basic AI tools because that's all they can access. Contrast that with a student from a well-off background, who can afford subscriptions to advanced models like GPT-4 and has top-notch devices at their disposal.

From the outset, the field is far from level. The student with access to better AI tools can generate higher quality work, iterate and refine it, and potentially game the system when it comes to things like detection tools. Meanwhile, the disadvantaged student is stuck with clunky, less sophisticated outputs.

Now, layer AI grading on top of that. If the grading AI is swayed by the polish and complexity of the work - which, as we've seen, it often is - then the student with better AI tools has an unfair advantage. They're likely to score higher grades, through no fault or merit of their own.

Over time, this can exacerbate achievement gaps and reinforce privilege. It's the classic "rich get richer" scenario, but with algorithms.

Flip it around, and consider what happens if schools, universities, and educators have access to differing levels of GenAI. What does it look like for a student whose work is assessed with GPT-3.5 compared to GPT-4? Efforts to "debias" AI have thus far only proven successful to an extent, and only for the more powerful models. For example, though it is far from perfect, GPT-4 exhibits less bias than GPT-3.5. So an educator or institution with the financial and technical resources to use a more powerful model will provide more sophisticated and potentially less biased feedback.

This is exacerbated by the fact that many educators and schools simply have not had the time to adjust to these technologies, and are being preyed upon by enterprising companies who aggressively push AI solutions. Many of these "AI for teachers" platforms build on top of GPT-3.5 or GPT-3.5 Turbo, as it would be more expensive to use GPT-4.

And that's only if the problem is resolved. A few months ago, Ryan Tannenbaum demonstrated the problem with GPT-3.5, 3.5 Turbo, and Claude 2. It's the same experiment that I replicated with GPT-4o, and will no doubt be replicable into the near future.

Who owns the contents of a student's brain?

I have huge issues with the cavalier approach to student intellectual property, extending back to before Generative AI was even in the mix. Plagiarism detection software, for example, is a multi-billion dollar industry built on top of student creativity and knowledge. I have never seen an education provider where students are permitted to "opt-in" to these platforms, and yet millions of pieces of their work are uploaded into the companies' databases daily. This has allowed these companies to grow and scale, with the volume of work compounding the usefulness and marketability of the software.

This problem is now exacerbated by AI. While plagiarism tools are no longer effective, it isn't stopping people from uploading student's IP into technology platforms. I've written about this before, but a lot of these AI tools are already built on pretty shaky ground when it comes to data provenance and consent. As educators, we have a duty of care to make sure student data isn't being misused or exploited.

So any use of AI in assessment needs to come with clear policies on data handling. No sharing student work with third-party AI providers without explicit, informed consent.

More appropriate uses of GenAI in assessment and feedback

While the use of GenAI for high-stakes, summative assessment is fraught with risks and limitations, as outlined in the previous sections, there are nevertheless several promising applications of this technology in education that warrant careful consideration and exploration. I also want to delineate between learning, assessment, feedback, and grading: these terms aren't interchangeable.

One area where GenAI can offer significant value is in providing low-stakes, formative feedback to students. By leveraging the natural language processing capabilities of models like GPT-4, educators can offer immediate, targeted feedback on specific aspects of student work, such as grammar, spelling, and punctuation. GenAI can also highlight areas for improvement in the structure, clarity, or coherence of a piece of writing, and offer prompts to encourage deeper reflection or analysis.

Crucially, however, such AI-generated feedback should be treated as a supplement to, rather than a replacement for, human feedback. As noted in my earlier post, Critic, Creator, Consumer, a balanced and thoughtful approach to GenAI involves recognising both its potential and its limitations. Educators should carefully review and contextualise the feedback provided by AI systems, and create opportunities for students to discuss, unpack, and apply that feedback in dialogue with their instructor.

Used judiciously, AI-generated feedback could encourage students to take greater ownership over their learning, engaging in self-assessment and revision of their work. It's also a workload-win. Teachers no longer have to take home piles of books and make grammatical or functional corrections that, in all likelihood, students will never read anyway.

GenAI has the potential to support adaptive learning and personalised support for students, though this is far from proven. By analysing patterns in student performance data, AI systems can identify specific skills or concepts that a student is struggling with, and recommend targeted resources or activities to address those individual learning needs: this is the basic premise of adaptive learning technologies including some of the features in Khan Academy's chatbot. This kind of "old school" AI based analytics could be coupled with GenAI to generate customised practice questions or prompts, adjusting the difficulty or complexity based on a student's demonstrated skill level. I'm yet to see this have a positive impact in practice, however.

As with all applications of GenAI in education, these adaptive learning systems must be designed and implemented with great care and attention to potential biases, limitations, and unintended consequences. They should be used to support and enhance, rather than replace, the role of human educators in diagnosing learning needs, building relationships, and guiding student growth.

Conclusions, but no clear answers

Fundamentally, the notion of using a language model (LLM) to grade student work is problematic, regardless of how sophisticated the prompt or input may be. While an LLM might generate feedback that appears thoughtful and thorough, it is essential to recognize that the model is producing a probabilistic output based on patterns in its training data, without the capacity for genuine reasoning or understanding. This holds true even if the input includes a detailed rubric, specific grading criteria, or sample student work with assessed comments. These additional contextual elements might help to anchor the LLM's response and make it seem more convincing, but ultimately, the output remains a product of statistical inference rather than true comprehension.

Arguing that "it's better if you use a better prompt" is a flawed premise because it fails to address the core limitations of LLMs in the context of evaluating student work. It's like suggesting that a spell checker can be used to assess the aesthetic qualities of a poem if given enough guidelines - while the spell checker may identify words that are commonly associated with poetic language, it lacks the appreciation for imagery, emotion, and figurative expression that a human reader would bring to the analysis.

In the same manner, an LLM may generate feedback that seems appropriate on the surface, but it lacks the deeper insights, critical thinking, and subjective judgment that a human educator brings to the grading process. We may be flawed, but at least we have the capacity to think, reason, and evaluate.

The use of AI in high-stakes assessment also raises concerns about fairness, accountability, and transparency: all key aspects of the Australian Framework for Generative AI in Schools along with other international education guidelines. While human graders may exhibit biases, they can be trained to recognise and mitigate them, whereas an LLM's biases are inherent to its training data and architecture, and may be more difficult to identify and address. The appearance of objectivity in AI-generated grades may mask underlying disparities and hinder efforts to ensure equitable evaluation. I would argue that using AI to generate results, numerical grades, or final, summative evaluations is actually contravenes the Framework

The use of LLMs for grading student work is a misapplication of the technology that fails to appreciate the fundamental differences between human and artificial intelligence. Rather than seeking to automate the assessment process, we should focus on leveraging AI to support and enhance human educators' abilities, while preserving the essential role of human judgment and expertise in evaluating student learning.

We need to begin by questioning why we grade work at all. Is assessment and grading simply a matter of competition, ranking, and placement? Do students actually need a letter or numerical grade at all? Or, as TEQSA asked recently in reference to the use of AI tools in assessment, are there other ways that students can demonstrate learning, and that educators can assess whether students have learned?

If nothing else, Generative AI is forcing us to have these difficult, sometimes uncomfortable conversations. Thanks to everyone who has joined in the discussion so far.

I regularly work with schools, universities, and faculty teams on developing guidelines and approaches for Generative AI. If you're interested in talking about consulting and PD, get in touch via the form below:

# data[].attributes.attachment_link.description
As a former secondary English teacher, senior examination assessor, and lecturer for initial teacher education, I understand the allure of using Generative AI (GenAI) for grading student work. We're all familiar with the workload of assessment and reporting. The idea of a tool that could save time and streamline the grading process is undeniably appealing....

# data[].attributes.headline
AI Education Podcast: Jason Lodge - Guiding Through Assessments

# data[].attributes.subject
AI Education Podcast: Jason Lodge - Guiding Through Assessments

# data[].attributes.attachment_link.readable_text
This week we continue our series on Assessment and AI. Ray talks with Jason Lodge from The University of Queensland, and who must have the longest business card in Australia, as he's Associate Professor of Educational Psychology in School of Education and Deputy Associate Dean in the Faculty of Humanities, Arts and Social Sciences!

The conversation talks about the challenges of assessment, and the options for rethinking assessment - and then we go deeper into Jason's views on the future of learning and assessment.

Jason's a great guest to share his experiences, as during 2023 he was on the TEQSA group of experts that came together to produce a report on assessment for Australian universities, Assessment reform for the age of artificial intelligence

https://www.teqsa.gov.au/guides-resources/resources/corporate-publications/assessment-reform-age-artificial-intelligence

Working on policy and guidance in an area where technology is developing so rapidly - and students are racing ahead of institutions, was interesting and Jason talks about the group dynamic. One of the interesting notes he talks about is the mindset: "The mantra we kept returning to is that we weren't trying to develop a map, but a compass. This is the direction we think we might need to head here."

# data[].attributes.attachment_link.title
AI Education Podcast: Jason Lodge - Guiding Through Assessments

# data[].attributes.attachment_link.description
This week we continue our series on Assessment and AI. Ray talks with from The University of Queensland, and who must have the longest business card in Australia, as he's Associate Professor of Educational Psychology in School of Education and Deputy Associate Dean in the Faculty of Humanities, Arts and Social Sciences!

# data[].attributes.headline
A scoping review on how generative artificial intelligence transforms assessment in higher education - International Journal of Educational Technology in Higher Education

# data[].attributes.subject
A scoping review on how generative artificial intelligence transforms assessment in higher education - International Journal of Educational Technology in Higher Education

# data[].attributes.attachment_link.title
A scoping review on how generative artificial intelligence transforms assessment in higher education - International Journal of Educational Technology in Higher Education

# data[].attributes.attachment_link.description
Generative artificial intelligence provides both opportunities and challenges for higher education. Existing literature has not properly investigated how this technology would impact assessment in higher education. This scoping review took a forward-thinking approach to investigate how generative artificial intelligence transforms assessment in higher education. We used the PRISMA extension for scoping reviews to select articles for review and report the results. In the screening, we retrieved 969 articles and selected 32 empirical studies for analysis. Most of the articles were published in 2023. We used three levels—students, teachers, and institutions—to analyses the articles. Our results suggested that assessment should be transformed to cultivate students’ self-regulated learning skills, responsible learning, and integrity. To successfully transform assessment in higher education, the review suggested that (i) teacher professional development activities for assessment, AI, and digital literacy should be provided, (ii) teachers’ beliefs about human and AI assessment should be strengthened, and (iii) teachers should be innovative and holistic in their teaching to reflect the assessment transformation. Educational institutions are recommended to review and rethink their assessment policies, as well as provide more inter-disciplinary programs and teaching.

# data[].attributes.headline
Menus, not traffic lights: A different way to think about AI and assessments

# data[].attributes.subject
Menus, not traffic lights: A different way to think about AI and assessments

# data[].attributes.attachment_link.readable_text
Earlier this year I went on a Disney cruise with my extended family. Seasickness aside, another notable experience was the dining: a sit-down dinner each night with a full menu - appetizers, soups, salads, mains, light mains, bread service, desserts, no-sugar desserts, after-tinis... This bewildering selection was supported by an enthusiastic and caring 'server' (kind of like a personal maître d') who got to know our family, our preferences, and our dietary needs - and was able to match these with expert recommendations from the menu.

It occurred to me while looking out across the great expanse of the Pacific Ocean and trying to hold in my latest meal that perhaps the way we approach students' use of generative AI in assessment isn't too far off this experience.
This article is an extension of a post that originally appeared on Linkedin.
Traffic lights

A common approach to AI in assessments has been to use a traffic light system. Red: can't use AI. Green: go for it. Yellow: only some forms of AI use are allowed. Sometimes there's a yellow-green, or yellow-red in the mix as well. This definitely brings familiarity and safety, which are much needed in a time of great turbulence. After all, traffic lights exude safety and inherently all of us educators are comfortable with requiring students to use their own brains for most of their work.

But, for take-home assessments that are not in some way supervised, we cannot know whether students use AI and cannot physically stop students from using it either. We also can't truly AI-proof assessments by making questions that AI "can't answer". The yellow light is not really valid here when it comes to unsupervised assessments, and neither is the red light. Just because we say an assessment should have restricted or inhibited use of AI doesn't mean that this will happen.

This leaves us really with 'go'.

Menus

When you sit down at one of the on-board restaurants on a Disney cruise and are welcomed by the effervescent server and presented with the overwhelming menu, you know you'll eat something (or you would have skipped it and gone to trivia instead). But what? This depends on your appetite, what others around you are eating, what is fresher on the menu, your dietary requirements, what you tried previously, etc. One of your server's roles is to help you make sense of the menu and help you choose the most suitable items for that point in time.

Sometimes you'll have an entrée and light main, sometimes you'll have the bread and five desserts, and sometimes you might have a soup, main, and a sugar-free dessert. There is no right or wrong (or allowed/disallowed), but there may be options that are more suitable or less suitable.

I think generative AI use in assessments is a bit like this.

There is a bewildering (and growing) array of generative AI tools available that may be applied for assessments. In addition to the staple ChatGPT, there are tools that help you summarise the literature (e.g. scite.ai, elicit.com), get answers from the internet (e.g. perplexity.ai, copilot.microsoft.com), produce images (e.g. DALL-E 3, MidJourney), create presentations (e.g. Gamma, Plus), analyse data (e.g. ChatGPT Plus), improve writing (e.g. Grammarly), produce videos of yourself talking (e.g. HeyGen), make music (e.g. Udio), and more. The mainly text-based mainstream generative AI tools (e.g. ChatGPT, Google Gemini, Microsoft Copilot) can themselves be used in myriad ways - brainstorming, checking, reflecting, editing, etc.

As educators who know our students and our teaching and assessment contexts, we are uniquely positioned to know what menu items to recommend. For any one assessment (and student, for that matter), there will be ways to use AI and tools that are more suitable or less suitable. Sometimes it will be more productive and responsible for students to use particular AI tools to engage with literature and edit their text. Other times it will be more suitable for students to other AI tools to provoke reflection, draft a structure, and then provide feedback on text. Want to order five desserts? Maybe not for this assessment, but it's a great idea for the next one.

Knowing what to order

We've been here before though.

Consider some standard tools: Word, Excel, PowerPoint, and Photoshop. When we set an assessment that asks students to create an annual report, we don't go around traffic-lighting particular tools. A good student will know to use Word for word processing and formatting, and Excel to calculate the tables of financial data needed. Another student may use tables in Word to lay out the financial data and work things out with a calculator - perhaps they were never taught how to use Excel. One student may use Photoshop to produce an impactful graphic for the front page, whereas another student may expertly use Powerpoint shapes to the same effect (this would be me).

The point here is that there is no right or wrong per se, but there are definitely tools (and ways to use them) that are more or less suitable. Some students will know, and some won't.

Our role as educators, like our friendly server who started to feel like part of the family by the 4th night of the cruise, is to help our students understand which tools (and ways to use them) are best suited to their own needs at a particular point in time for a particular assessment. This is a key part of developing students' information and digital literacy skills.

An 'AI for assessment' menu

Brainstorming with my team (the wonderful Jessica Frawley, Samantha Clarke, Eszter Kalman, Benjamin Miller, and Robyn Martin), we came up with some sort of menu of AI for assessments. Each of these menu items are supported by different AI tools, used in different ways.
As a critical friend - Soups

Suggest analyses

Provoke reflection

Provide study/organisation tips

Practicing

Getting started - Entrees

Suggesting structure

Brainstorming ideas

Engaging with literature - Bread service

Suggesting search terms

Performing searches

Summarising literature

Identifying methodologies

Explaining jargon

Fixing reference list

Generating content - Mains

Writing some text

Making images, video, audio

Making slidedecks

Analyses - Lighter mains

Performing analyses of data, text

Suggesting counterarguments

Editing - Coffees

Editing tone

Improving clarity and readability

Fixing grammar

Shortening

Feedback - Desserts

On all of the above elements

Specifically on rubric criteria

If you'll excuse my poor attempt to connect these to food, hopefully this menu helps to illustrate the nature of generative AI for assessments. For a particular assessment, for example, we may know it will be more beneficial for student learning to use AI predominantly as a critical friend, to engage with literature, and then to edit. Sure, for that assessment they could use AI to generate some content or analyse data, but this will degrade students' attainment of the learning outcomes. For another assessment and another set of learning outcomes, it may be quite palatable and beneficial to order bread, a main, and skip the AI dessert and do that oneself.

Many of these 'AI for assessment menu' items can be achieved through text-based generative AI tools like ChatGPT, Copilot, Gemini, and Claude. It really depends on how you prompt it. Some of these could be built as AI 'agents' that are designed by educators to perform a certain task like provoke reflection, provide rubric-specific feedback, or suggest counterarguments.

As an example, for the unit that I'm teaching into this semester, these are the recommendations from the menu:

Assessment 1 (self reflection about current challenges in a practice setting)

Soup - AI to provoke reflection - we have built a Cogniti AI agent that was instructed to help students reflect on key issues in the context of the frameworks used in the unit

Dessert - AI to provide feedback based on rubric criteria - we built another Cogniti AI agent that was given the marking criteria and assignment brief, and instructed to provide feedback to students before submission

Assessment 2 (annotated bibliography)

Entree/bread service - AI to brainstorm search terms and ideas - we built another Cogniti AI agent to help students unpack unfamiliar terminology and suggest potential search terms for literature databases

Bread service - we showed our students scite.ai, elicit.com, and Research Rabbit and explained the ins and outs of each of these tools, and the importance of deep-diving into the literature once key papers were identified

Dessert - AI to provide feedback based on rubric criteria - again, another Cogniti AI agent that was provided the rubric and assignment brief

We have yet to consider what menu items are most appropriate for assessment 3 (group project with forward plan)

As educators, we know the contexts of our units. We know the learning outcomes, and the assessments, and how students might best learn. We know our students, their interests, and their needs. As enthusiastic and caring educators, let's also come to know generative AI and the ways that it can be used, so that we can help students steer clear of less suitable choices and help them embrace choices that will not only be palatable, but productive and responsible.

# data[].attributes.attachment_link.title
Menus, not traffic lights: A different way to think about AI and assessments

# data[].attributes.attachment_link.description
Earlier this year I went on a Disney cruise with my extended family. Seasickness aside, another notable experience was the dining: a sit-down dinner each night with a full menu - appetizers, soups, salads, mains, light mains, bread service, desserts, no-sugar desserts, after-tinis...

# data[].attributes.headline
What could AI-assisted assessment look like?

# data[].attributes.subject
What could AI-assisted assessment look like?

# data[].attributes.attachment_link.readable_text
Over the past few months, I've used generative AI to help me scope ideas and scaffold research funding applications. With the support of AI, I've identified research gaps and potential projects that could fill them. I've fine-tuned a multilevel guided prompt that supports me in thinking critically about the purpose, aims, research questions, methods and methodologies.

This process is similar to one of the assessments our students submit, which requires them to identify a problem in practice or a challenge that will be met by a practice-based intervention. Currently, this is a formative assessment involving a collaborative oral element where the peer groups read each other's project proposals and then provide feedback to each other in a synchronous online meeting. This works reasonably well and is authentic in its purpose and outcomes; however, sometimes, the feedback isn't as critical as required.

Reflecting on this with students, we've identified two main hurdles. First, the discomfort associated with providing critical feedback to peers, and second, the struggle to provide feedback on unknown territory - the classic conundrum of not knowing what you don't know. This is particularly challenging in a 'live synchronous' assessment. The existing project scope assessment aims to evaluate the critical understanding of applied research methods and tools within the context of practice-based change projects, as well as how the students critically evaluate all ethical considerations and mitigations relevant to practice-based change projects.

Drawing inspiration from the University of Sydney's innovative 'two-lane' approach to assessment, I've started reimagining our assessments to better equip our students for an AI-integrated future. In the model proposed by Danny Liu and Adam Bridgeman, 'Lane 1' assessments ensure students have mastered the skills and knowledge our programs require, while 'Lane 2' assessments motivate students to learn and teach them to engage responsibly with AI.

I often encourage our students to use generative AI to support the development of research ideas. I have written about this in the past, and based on my experimentation with AI for developing research proposals, the project scope assessment seemed the ideal opportunity to introduce a 'lane 2' generative AI approach. I could also see the potential of mitigating some of the shortcomings identified in student feedback about the assessment, particularly the challenge of providing critical feedback to peers. Devoid of feelings (as it so often lets us know!), AI can serve as an impartial 'critical friend', providing feedback that students may not have considered. This unique characteristic of AI, its lack of emotions, makes it easier for students to critique. I'm hoping that students can freely express their thoughts and criticisms without the fear of offending or hurting someone's feelings. The aim is that this can build confidence in this area and, in turn, support critical feedback for real human peers!

The revised 'lane 2' project scope assessment encourages students to input their emerging project ideas into the generative AI using the provided multi-staged mentoring prompt. The generative AI then attempts to provide critical feedback and suggests improvements based on the parameters, in this case, learning outcomes written into the prompt. This process culminates in the synchronous online meeting where students collaboratively reflect on their original scope and the version with modifications suggested by the generative AI. They justify their critique against a framework based on the assessment outcomes, demonstrating the critical understanding and evaluation we aim to foster, and in doing so, they collaboratively navigate the lanes of learning in an AI-integrated world.

Another intentional outcome of this 'lane 2' approach is to raise AI literacy. While some students were using AI to support the development of their research ideas, others were avoiding it because they feared ' being caught cheating ' or because they did not understand how to prompt effectively. By making generative AI use an integral part of the assessment and by reflecting on the output collaboratively, the students understand use cases for generative AI as well as the potential shortcomings of the tools.

In the revised assessment, I've intentionally included a comprehensive, multi-staged mentoring prompt designed to guide a student through the detailed process of developing a research proposal for a practice-based research project. It provides a structured and interactive framework where the AI acts as a mentor, offering critique and suggestions and ensuring the student is prepared before moving to the next stage. This method ensures thorough coverage of each aspect of the project scope, facilitating deep engagement and iterative refinement. This also provides a consistent experience for all learners and supports them with a framework for creating their own prompts. I've also indicated they are free to tinker with the prompt and make it work for their context. Further into the course, I will encourage students to create their own prompts and support critical evaluation based on the rubric suggested by Danny and Adam.

At this stage, we have tested the prompt using the generative AI models available to our students and trialled the concept with a small group of students. In the next iteration of the course, we will introduce the assessment with the AI prompt and aim to quantify the level of critical thinking based on a modified scale relevant to generative AI use. If anyone has any suggestions, it would be great to learn more!

A link to the prompt I developed can be found here. I'm sure there could be further refinements, so I'll keep working on it. Any feedback would be appreciated.

# data[].attributes.attachment_link.title
What could AI-assisted assessment look like?

# data[].attributes.attachment_link.description
Combining a critical friend and peer review to raise critical thinking, research quality and AI literacy

# data[].attributes.headline
Embracing the future of assessment at the University of Sydney

# data[].attributes.subject
Embracing the future of assessment at the University of Sydney

# data[].attributes.attachment_link.readable_text
In an era where generative artificial intelligence (AI) is reshaping the educational landscape, Sydney is pioneering a refreshed approach to assessment and feedback. This initiative is not just about keeping pace with technological advancements but also about enhancing the quality and integrity of our programs. Here, we explore the opportunities and principles underpinning this and crucial shift.

A 'two-lane' approach to assessment

We are taking a 'two-lane' approach to assessment design in the age of AI:

: These are designed to ensure students have mastered the skills and knowledge our programs require. They are supervised and designed for the 'assessment 'Lane 1' assessments of learning'.

'Lane 2' assessments : These motivate students to learn, and teach them to engage responsibly with AI. They focus on productive and responsible participation in an AI-integrated society.

Most of our assessments should be in 'lane 2' - that is, they should be assessments that engage students in authentic learning.

This model ensures that we meet Australian Government's Tertiary Education Quality and Standards Agency (TEQSA) guidance, advocating for inclusive, contextualised, and trustworthy assessments and preparing students for an AI-infused world.

The University's refreshed assessment principles

We have updated our assessment principles to align with contemporary technologies, best practices, and student skill development with a strong focus on providing students with useful and timely feedback to help them succeed.

Assessment practices must promote learning, evaluate outcomes and facilitate reflection and judgement. There is renewed emphasis on assessments that promote and measure learning, supported by constructive and timely feedback.

Assessment practices must be clearly communicated. There is additional emphasis on helping students appreciate how assessment tasks are related to learning outcomes.

Assessment practices must be inclusive, valid and fair. This reiterates our commitment to standards-based assessment, and provides additional impetus to moderate standards to ensure inclusivity and equity.

Assessment practices must be regularly reviewed. New emphasis is placed on keeping assessments up to date in alignment with new technology (including AI), and including the student voice in evaluating assessments.

Assessment practices must be integrated into program design. This newly developed principle introduces the integration of assessment and feedback across the program, supporting student development from enrolment to graduation. There is an emphasis on validation of students' attainment at various points across a program, through trustworthy and coherent assessment tasks. This principle also reminds us to equip students with skills for their studies and the future, including with appropriate technologies.

Assessment practices develop contemporary capabilities in a trustworthy way. This newly developed principle focuses on enabling students to develop and demonstrate skills with relevant technologies, whilst ensuring academic integrity. It emphasises the role of assessment practices in helping students build the ability to work ethically with technologies like AI, through assessments that may be unsupervised and serve to motivate and drive learning ('lane 2' assessments). It also emphasises the importance of having other assessments that can assure learning, across a program ('lane 1' assessments).

Helping to understand how we are going

To help Associate Deans (Education) and other staff keep track of and support assessment redesign, two new fields will appear for each assessment item in Sydney Curriculum.

A selection in the new dropdown field is required, and if you genuinely have not had a chance to consider this, we encourage you to select the 'Not considered' option so that the appropriate supports can be made available.

There are various combinations of these two fields:

This is a typical 'lane 1' assessment for the 'assessment Secured assessment, generative AI not allowed: of learning'. Optimally, these assessments would be designed at the program level and serve to confirm attainment of program learning outcomes. These assessments should be used sparingly and strategically. They may involve assessment types such as tests/exams, viva voces, or interactive oral assessments.

Secured assessment, generative AI is allowed: This can also be a 'lane 1' assessment, but less common. In some disciplines, we may need to assure that students can use particular generative AI tools under supervised conditions. For example, architecture programs may need to assure that students can use image-based generative AI tools.

Unsecured assessment, generative AI not allowed: This combination is invalid, because there is no reliable way to prevent or detect the use of generative AI in an assessment that is not supervised.

most of our assessments. They are 'assessment Unsecured assessment, generative AI is allowed: This is a typical 'lane 2' assessment. These should comprise for learning', and serve to motivate and engage students in learning through the process of assessment. We should encourage and support the appropriate, responsible, and productive use of AI in these assessments.

Supporting you to reimagine assessment and feedback

To facilitate these changes, the University offers numerous resources:

: Module 05 now includes content on AI challenges and opportunities in assessment. University of Sydney staff can Modular Professional Learning Framework (MPLF) access the MPLF on Canvas.

: Stay informed about AI developments and their potential in enhancing your teaching methods. We will be adding more of these in the new year to our Workshops on AI and assessment design events page.

: Educational designers are available for one-on-one sessions to refine your assessment and feedback strategies. University of Sydney staff can 1:1 consultations access these consults here.

Within our faculties and schools, we are also fostering a network of champions to assist in evaluating and evolving current assessment practices.

Moving forward together

The path ahead is both challenging and exciting. By working together, we can adapt to the evolving environment and lead the way in innovative and effective teaching and assessment practices. Tackling this challenge will allow us to ensure our students are equipped for success in an AI-integrated future.

# data[].attributes.attachment_link.title
Embracing the future of assessment at the University of Sydney

# data[].attributes.attachment_link.description
In an era where generative artificial intelligence (AI) is reshaping the educational landscape, Sydney is pioneering a refreshed approach to assessment and feedback. This initiative is not just about keeping pace with technological advancements but also about enhancing the quality and integrity of our programs. Here, we explore the opportunities and principles underpinning this and crucial shift.

# data[].attributes.headline
Responding to generative AI for assessments University of Sydney

# data[].attributes.subject
Responding to generative AI for assessments University of Sydney

# data[].attributes.body
semester 2, 2023

# data[].attributes.headline
What to do about assessments if we can't out-design or out-run AI?

# data[].attributes.subject
What to do about assessments if we can't out-design or out-run AI?

# data[].attributes.attachment_link.readable_text
Generative AI is improving at a rapid pace and is increasingly integrated into software that we use daily. It is becoming increasingly clear that there isn't going to be a way to design take-home assessments that are 'AI-proof'. Assessments that are authentic, or reflective, or personal, or contextualised, or multimodal, or localised - these can all be completed by new generations of generative AI. Perhaps this isn't all bad news - human-AI collaborations are likely to be the norm in the near future, so we need to start thinking about how to embrace AI as part of assessment. But, we still need to assure that students have achieved learning outcomes - how do we do this, whilst embracing AI?

Together with folk across the University of Sydney, we put together a quick guide to help coordinators ensure that assessment changes have longevity, even as AI progresses. The guide also emphasises the opportunities for us to reflect on the human side of teaching and learning, help students become better prepared for AI-augmented careers, and encourage the ethical, accountable, and transparent use of AI. Here is a summary of the contents of the guide; you are welcome to download the full guide. Note that the guide has been produced for use within the University of Sydney, but it has broader applicability across different institutions.
Download the full guide
Diagnosing and addressing impact

What are the immediate actions you need to take for your assessments?

Discover the capabilities. Run your assessment prompt(s) through generative AI, preferably using the more powerful models available (such as GPT-4 for text generation, Adobe Firefly for image generation, etc). Ensure you play around with the prompts to properly explore AI's potential capabilities (see Appendix 1 in the full guide).

Evaluate student motivations. Students are more likely to undertake assessment with integrity if they feel:

Autonomy: having real choice about topic and mode, and seeing how the assessment meaningfully connects with their life and career.

Competence: being supported to build confidence and skills gradually.

Relatedness: feeling connected to teachers and peers and that they matter.

Balance assurance of learning and use of AI. All units need to have assessments that assure student attainment of learning outcomes - this is likely best performed in live, supervised settings. It's also critical to ensure we help students use AI productively and responsibly, which can be done by redesigning other assessments to address appropriate learning outcomes. The table on the next page provides guidance for balancing these.

Reduce the perceived workload and pressure. Assessments with clear instructions and criteria, have meaningful and appropriate challenge, that provide sufficient time for completion, and which help students develop confidence in their abilities (e.g. through structured drafts and feedback) will lead to more positive academic integrity outcomes.

Decide and communicate. It's important to differentiate AI use for learning, and AI use for assessment. Use this guide to determine what level of AI usage is appropriate in context, and clearly communicate this to students - including situations where AI use is not appropriate. Examples of appropriate use and wording about how to acknowledge AI use are provided in the full guide.

The full guide has specific advice for different assessment types including take-home assignments, small continuous assessments, groupwork assessments, and exams and tests.

Redesigning assessments

Some level of assessment redesign is required across almost every unit to both manage the risk of generative AI and provide students with opportunities to engage with it productively and responsibly. In a world where AI is inescapable, assessments should both assure learning in secure settings, and adapt to the reality of AI in other settings, as appropriate to each discipline.

A 'two-lane' approach

The two-lane approach below emphasises balance between assurance, and human-AI collaboration. The reality in any one unit will likely be a situation where some assessments lie in lane 1 in order to assure attainment of all learning outcomes, but most other assessments lie in lane 2. Fundamentally, we want to develop students who are well-rounded and can contribute and lead effectively in authentic, contemporary environments (which will include AI), and also be assured of their learning. Therefore in this context, it is important to privilege lane 2 assessments with a higher weighting than lane 1 assessments.

We do not foresee a viable middle ground between the two lanes. It needs to be assumed that any assessment outside lane 1 (i.e. that is un-secured) may (and likely will) involve the use of AI.

Lane 1 - Examples of assured 'assessment of learning'

In-class contemporaneous assessment e.g. skills-based assessments run during tutorials or workshops

Viva voces or other interactive oral assessments

Live simulation-based assessments

Supervised on-campus exams and tests, used sparingly, designed to be authentic, and for assuring program rather than unit-level outcomes

Lane 2 - Examples of human-AI collaboration in 'assessment as learning'

Students use AI to suggest ideas, summarise resources, and generate outlines/structures for assessments. They provide the AI completions as an appendix to their submission.

Students use AI-generated responses as part of their research and discovery process. They critically analyse the AI response against their other research. The AI completion and critique provided as part of the submission. Appendix 3 in the full guide provides suggestions on how to assess this.

Students initiate the process of writing and use AI to help them iterate ideas, expression, opinions, analysis, etc. They document the process and reasoning behind their human-AI collaboration. The documented process demonstrates how the collaborative writing process has helped students think, find their voice, and learn. The documented process is graded and more heavily weighted than the artefact. Appendix 3 in the full guide provides suggestions on how to assess this.

Students design prompts to have AI draft an authentic artefact (e.g. policy briefing, draft advice, pitch deck, marketing copy, impact analysis, etc) and improve upon it. They document the process and reasoning: initial prompt, improvements, sources, critiques. The documented process demonstrates learning, is graded, and is more heavily weighted than the artefact. More information. Appendix 3 in the full guide provides suggestions on how to assess this.

We do not foresee a viable middle ground between the two lanes. It needs to be assumed that any assessment outside lane 1 (i.e. that is un-secured) may (and likely will) involve the use of AI.

An example of assessments across both lanes

In this example, students need to apply marketing strategy concepts in real-world scenarios, demonstrate their communication skills, and evaluate the effectiveness of different marketing strategies.

The lane 2 assessment might involve students collaborating with AI such as Bing Chat (which is internet-connected) to perform market research and competitor analysis, and other AI such as Adobe Firefly for the visual elements of campaign design. Students document their interactions with the AI tools, including the AI's initial market research and analysis and their critique and fact-checking processes to evaluate the AI's outputs. Students also critique whether AI provided novel insights and whether it missed critical factors. This is then presented live in class. The grading of the assessment is more heavily weighted on the documented process of critical co-creation (see Appendix 2 and Appendix 3 of the full guide).

The corresponding lane 1 assessment might involve a live Q&A after the presentation, where students need to defend their research and analysis through targeted questions. This can be made to simulate real-world business meetings, and helps to assure that students have met their learning outcomes of applying marketing strategy concepts and evaluating effectiveness of marketing strategies. Another lane 1 assessment might involve giving students an unseen case study of a company that has recently launched a new product; in a live, supervised setting, they need to evaluate the effectiveness of the marketing strategy and propose areas of improvement.

Using AI as part of assessment

The full guide provides examples of uses of AI in assessment, as well as wording that could be presented to students to explain this. It also provides examples of how students might acknowledge the use of AI.

Some examples include:

Generating ideas for assessment: You may use AI tools such as <ChatGPT, Bing Chat, and other generative AI> to <brainstorm ideas and approaches> for completing your assignment.

Creating media for assessment: You may use AI tools such as <DALL-E, MidJourney, Stable Diffusion, Adobe Firefly, and other image generative AI> to generate <images> that you use as part of your submission.

Providing feedback on work: You may use AI tools such as <ChatGPT, Bing Chat, and other text-to-text generative AI> to seek feedback on your written work.

Searching literature: You may use AI tools such as <elicit.org, perplexity.ai, and researchrabbit.ai> to find and summarise research articles. You then need to incorporate the scholarship yourself into your submission.

Detecting students' AI use

Software that purports to detect the use of AI-generated text is prone to false positives and false negatives. Research has suggested these tools are not reliable and may be biased against non-native English writers. In addition, ChatGPT does not 'know' whether it generated a piece of text - even if it may produce a convincing response when asked.

Our advice is that you should not submit students' work to AI detection software yourself - this is potentially a breach of student privacy and intellectual property.

More concrete advice

Testing your own assignment with generative AI

It's important to test your assignment(s) against generative AI to gauge what kinds of outputs could be produced. When prompting generative AI, it's important to remember that better prompts will yield better results - don't stop at the first prompt/response and dismiss AI if its initial response is not impressive.

Appendix 1 of the full guide has a step-by-step guide on how to properly try out your assignment prompts on AI.

Suggestions for rubric criteria that target higher order thinking skills

A marking rubric is a tool that allows teachers, markers and students to form a shared understanding of the specific criteria and standards used to make academic judgements. A rubric directs students' work by providing descriptions of the standards at different levels of achievement. Appendix 2 of the full guide has some sample wording that you should adapt for your needs.

Note that the rubric suggestions attempt to privilege the more human elements of writing and composition as part of assessment design and grading. It is becoming trivially easy for AI to replicate these elements if prompted the right way. Therefore, changing your rubric should not be the only change you make to assessment in response to AI.

Suggestions for rubric criteria that privilege the process of human-AI collaboration in assessment rather than the product

Appendix 3 in the full guide provides rubric examples that can be adapted for assessments that involve human-AI collaboration. These rubric criteria are designed to help you assess the process of learning and evaluate whether students have appropriately developed and applied disciplinary skills and knowledge when they are working with AI.

Approaches to viva voces, live Q&A, and other interactive oral assessment

Interactive oral assessments can be an authentic, secure, and engaging way to assess attainment of learning outcomes. Optimally, they are conversational in nature, as opposed to a question and answer oral test. They allow you to probe deeper understanding - it is often very easy to spot a student who doesn't understand a concept by their oral responses. We have provided guidance on these assessment tasks, and provide additional guidance below for interactive oral assessments in the context of AI.

Appendix 4 in the full guide has more guidance about these assessments.

Tell me more

# data[].attributes.attachment_link.title
What to do about assessments if we can't out-design or out-run AI?

# data[].attributes.attachment_link.description
Generative AI is improving at a rapid pace and is increasingly integrated into software that we use daily. It is becoming increasingly clear that there isn't going to be a way to design take-home assessments that are 'AI-proof'.

# data[].attributes.headline
The Artificial Intelligence Assessment Scale (AIAS): A Framework for

# data[].attributes.subject
The Artificial Intelligence Assessment Scale (AIAS): A Framework for

# data[].attributes.body
Ethical Integration of Generative AI in Educational Assessment

# data[].attributes.attachment_link.description
Checks assessment tasks and recommends a level of the AIAS

# data[].attributes.headline
Assessment of student learning is broken (opinion)

# data[].attributes.subject
Assessment of student learning is broken (opinion)

# data[].attributes.headline
Conférence 1 – L’IA en éducation: quand les colonnes du temple tremblent,

# data[].attributes.subject
Conférence 1 – L’IA en éducation: quand les colonnes du temple tremblent,

# data[].attributes.attachment_link.title
Conférence 1 – L’IA en éducation: quand les colonnes du temple tremblent, Florian Meyer et Isabelle Nizet, 6 décembre 2023, 17h

# data[].attributes.attachment_link.description
L’IA en éducation : quand les colonnes du temple tremblent…En éducation, le recours aux outils d’intelligence…

# data[].attributes.headline
10 stratégies pour éviter le plagiat IA dans evaluation

# data[].attributes.subject
10 stratégies pour éviter le plagiat IA dans evaluation

# data[].attributes.headline
Guide rethinking assessment at the age of AI

# data[].attributes.subject
Guide rethinking assessment at the age of AI

# data[].attributes.headline
The AI Assessment Scale in Action: Examples from K-12 and Higher Education Across the World

# data[].attributes.subject
The AI Assessment Scale in Action: Examples from K-12 and Higher Education Across the World

# data[].attributes.attachment_link.title
The AI Assessment Scale in Action: Examples from K-12 and Higher Education Across the World

# data[].attributes.attachment_link.description
The AI Assessment Scale has already been adapted in many different contexts, both K-12 and higher education, and it’s been incredibly positive to see the impact that it has had in education a…

# data[].attributes.headline
Join in the conversations about stimulating examples of integrating Artificial Intelligence (AI) into assessment - Monash Teaching Community

# data[].attributes.subject
Join in the conversations about stimulating examples of integrating Artificial Intelligence (AI) into assessment - Monash Teaching Community

# data[].attributes.attachment_link.readable_text
Join in the conversations about stimulating examples of integrating Artificial Intelligence (AI) into assessment
By Joanne Tanner & Ari Seligmann
Posted Wed 13 March, 2024
We are operating in an era of unprecedented and rapid transformations, propelled by emerging technologies, such as Artificial Intelligence (AI). Advances and new technologies seem to appear daily, but where can one find reliable examples to judge the potentials of AI for higher education?

Educators regularly rely on a combination of 1) precedents and ideas that have been attempted before and may offer some lessons and cautions 2) verifiable evidence from peer-reviewed academic literature and 3) established theories and ideas that can guide decisions and ground our practices. Evidence-based development, building on solid pedagogical foundations, provides confidence and direction, but evidence takes time to collect.

Where might we find some examples of AI being responsibly used in education in order to spark our imaginations, offer some suggestive models, and provide some confidence and hope for integrating AI into our own pedagogical practices?

The speed of disseminating precedents and academic publications rooted in verifiable evidence is lagging behind the fast paced development and release of new AI tools and approaches. So where might we find some examples of AI being responsibly used in education in order to spark our imaginations, offer some suggestive models, and provide some confidence and hope for integrating AI into our own pedagogical practices?

A quick internet search may yield some clues as many around the world scramble to create useful guidance to inform directions. For example, explore 13 ways you could integrate AI tools into teaching, 101 creative ideas to use AI in education, a crowdsourced collection or UNESCO guidance on possible roles, Stanford's consideration of the pedagogical uses of AI chatbots or UK National Center for AI insights. You may even try asking for recommendations from your favourite Large Learning Model (LLM) chatbot (Claude, Gemini, Bing Copilot, ChatGPT, etc.) and cautiously consider the suggestions.

Another avenue may be to review the range of examples recently compiled by members of the Monash AI in Education Learning Circle. The Circle is a dedicated group of colleagues who spent the past few months thinking through the range of complex issues and collecting examples of AI use in assessments to share. Examples by and for Monash colleagues who are familiar with our context.

Co-author and contributor Joanne Tanner describes our efforts to bring together practitioners and proponents of AI for education from across all Faculties at Monash to raise awareness and provide guidance for our colleagues.

Our first focus was on AI in Assessments. We tried to remain discipline-agnostic and provide guidance to engage and assist anyone who is interested in using AI in their assessments, but also those who may wish to limit or preclude its use for pedagogical reasons. We have collected over 15 examples showcasing a broad range of AI tools and use cases to make sure that no matter what discipline or what type of assessment you are planning, there will be something there that you can use for inspiration. The resources cover a range of currently available AI tools offering another avenue for educators to explore.

More than recipes to repeat, we tried to find frameworks to think through the examples of AI in assessment.

More than recipes to repeat, we tried to find frameworks to think through the examples of AI in assessment. The collection offers some prototypes that identify core issues and can be modified for diverse pedagogical contexts. No matter your stance or purpose, be it adding AI to a current assessment, creating a new kind of assessment, or carefully ensuring that AI is restricted to ensure demonstration of AI-independent human knowledge and skills, we believe you will find something of value in the collection of examples.

Explore the range of examples gathered from across Monash, which you can sort by categories that suit your interests or needs. However, this collection is simply a few contributions to ongoing conversations, and we invite everyone to post their comments, including thoughts and further examples of designing AI into or out of their assessments.

Joanne Tanner

Ari Seligmann

# data[].attributes.attachment_link.title
Join in the conversations about stimulating examples of integrating Artificial Intelligence (AI) into assessment - Monash Teaching Community

# data[].attributes.attachment_link.description
By Joanne Tanner & Ari SeligmannPosted Wed 13 March, 2024 We are operating in an era of unprecedented and rapid transformations, propelled by emerging technologies, such as Artificial Intelligence (AI). Advances and new technologies seem to appear daily, but where can one find reliable examples to judge the potentials of AI for higher education?