# data[].attributes.body
looking for examples

implications for teachers

How should teachers create learning oppurtunities that foster learning using the aspect of multimodality?

# data[].attributes.headline
Thesis question: How does multimodality influence the way individuals learn language? How does multimodality influence the school learning context?

# data[].attributes.subject
Thesis question: How does multimodality influence the way individuals learn language? How does multimodality influence the school learning context?

# data[].attributes.body
The term multimodality refers to an individual’s use of different modes (i.e. channels of communication) for the purpose of conveying meaning (Kessler 2022: 551)

# data[].attributes.body
introduced by Halliday’s (1978) work involving social semiotics, which is concerned with understanding the different modes of communication that humans use (Kessler 2022: 551)

# data[].attributes.body
Halliday argued that researchers and instructors should go beyond privileging sentence- and grammatical-level constructions (Kessler 2022: 551)

# data[].attributes.body
Halliday wanted to shift their attention to focusing on how people use a combination of modes in different contexts for different purposes (Kessler 2022: 551)

# data[].attributes.body
modes: linguistic, visual, aural, gestural, or spatial in nature (Kress 2003) (Kessler 2022: 551)

# data[].attributes.body
modes of expression can be visual (e.g., drawing, painting, video), print (e.g., books, newspapers, environmental print), gestural (e.g., miming, pointing, acting out), dramatic (e.g., role-playing, improvisation, formal acting), and oral (e.g., informal talk, public speaking) modes (Kress 2000; Stein 1998). (in Rowsell & Collier 2017: 313)

# data[].attributes.body
working out how the components – or modes – work together to make meaning, and understanding in what ways particular modal choices are shaped by the interpersonal, social and cultural contexts of their use (Flewitt, Price & Korkiakangas 2019:3)

# data[].attributes.body
studies how people behave and interact in contemporary life (Flewitt, Price & Korkiakangas 2019:4)

# data[].attributes.body
focus on performance: study of language use

# data[].attributes.headline
moving away from the structural approach

# data[].attributes.subject
moving away from the structural approach

# data[].attributes.body
--> language as a set of formal structures which are studied separately from how they are use

-->not seen as an instrument of communication

# data[].attributes.body
the dynamic system is constantly changing and being restructured because of how it is used

rules and patterns are not fixed but can be changed or recognized constantly

depending on how and in what context the language is used

# data[].attributes.body
domain-general cognitive processes:

Processing language is connected with a range of cognitive abilities and experiences

# data[].attributes.body
looks at studies in Conversation Analysis (CA) (Flewitt, Price & Korkiakangas 2019:4):

to examine the structure and organisation of people’s communication practices

considering how bodies are also used in interactions, including a focus on gaze and gesture

# data[].attributes.body
each with a specialized role, to meaning-making; hence relations among modes are key to understand every instance of communication. (Adami 2016:2)

# data[].attributes.body
What constitutes a mode depends on the social group who uses it and the range of meanings that the group can express through its resources. (Adami 2016:13)

# data[].attributes.body
in face-to-face communication between typical hearing adults, language is multimodal (Morgenstern, 2022: 2)

# data[].attributes.body
multimodal information structures infants' comprehension and production (they extract meaning from multimodal cues)

(Morgenstern, 2022: 2)

# data[].attributes.body
children demonstrate their skills to resort to multiple semiotic resources in their daily interactions and expertly use them according to their expressive needs

# data[].attributes.body
usage-based approaches take into account the language that surrounds children or that is addressed to them. (Morgenstern, 2022: 3)

-->children internalize conventions in parallel to other cognitive and social skills, such as the ability to follow the others' gaze, to attract their attention, to read their intentions, the ability to make analogies, to categorize, to symbolize.

# data[].attributes.body
children participate in exchanges which are strongly scaffolded (Bruner, 1975) by their care-givers. (Morgenstern, 2022: 4)

# data[].attributes.body
adults adapt to the child's motor, cognitive, and linguistic development

adults verbalize every possible visual, vocal, actional, tactile cue enacted by the child to co-create dialogue (Morgenstern, 2022: 5)

# data[].attributes.body
instances of “social referencing” (Klinnert, 1983) constitute “affective frames” (affective display, especially through facial expressions) (Ochs & Schieffelin, 1989) that are fundamental to children's cognitive and linguistic development (Morgenstern, 2022: 5)

# data[].attributes.body
Child-adult interactions: flexibility of the multimodal linguistic system can be grasped

-->adults constantly adjust to their children according to their span of attention and their age

# data[].attributes.body
Prosodic features regulate infants' attention, make linguistic input more salient to them, facilitate their engagement, and helps them grasp emotional signals (Fernald, 1992; Saint-Georges et al., 2013).

ChatGPT prompt: What is Prosody (linguistics)?

Prosody in linguistics refers to the rhythm, stress, and intonation of speech. It encompasses the patterns of pitch, loudness, tempo, and rhythm in spoken language, which help to convey meaning, emotion, and grammatical structure.

# data[].attributes.body
their attention span and their language comprehension abilities are more limited --> infants are more sensitive to prosodic cues and to the voice quality (Morgenstern, 2022: 7)

# data[].attributes.body
Multimodal cues produced by adults (gazing, pointing, touching, manipulating) seem to facilitate word learning (Booth et al., 2008)

--> infant's selective attention can be positively affected. facilitation of children's comprehension--> symbolic gestures, especially pointing (Iverson et al., 1999)

# data[].attributes.body
o   gesturing allows children to express more specific semantic functions and thus communicate meanings (Morgenstern, 2022: 8)

o   a tool to expand both their lexicon and their communicative repertoire (Özçalıs¸kan & Goldin-Meadow, 2005). (in Morgenstern, 2022: 8)

# data[].attributes.body
using storyboard activities, picture description tasks, flashcards, audio- and video-based files, multimedia projects (Kessler 2022: 551)

# data[].attributes.headline
5. number and range of multimodal tasks in academia

# data[].attributes.subject
5. number and range of multimodal tasks in academia

# data[].attributes.body
--> possessing multimodal literacy is considered crucial

-->for example, humanities produce portfolios whereas engineering and sciences create lab reports (Kessler 2022: 552)

# data[].attributes.headline
6. impact of multimodality on L2 development 

# data[].attributes.subject
6. impact of multimodality on L2 development 

# data[].attributes.body
o   no effects: Cho and Kim (2021) found no significant differences in measures of quality, content, or language when comparing the outcomes of groups who engaged in monomodal versus multimodal composition tasks. (Kessler 2022: 552)

o   Positive effects: Vandommele et al. (2017) found that learners who participated in collaborative multimodal writing assignments outperformed a control group in terms of their texts’ perceived communicative effectiveness, lexical diversity, text length, and other measures. (Kessler 2022: 552)

-->multimodal assessments can support students’ learning by situating the academic content within the lived experiences of students.

Multimodal assessments can

provide students with diverse receptive and expressive opportunities

position students to take on critical stances

allow children to demonstrate diverse assets

growth in language and literacy (Lawrence & Mathis, 2020) (Lim, Toh & Nguyen 2022:9).

o  both linguistic and non-linguistic modes in classrooms develop learners’ communicative competence (Kessler 2022: 552)

o  non-linguistic modes are of equal value, since charts, figures, graphs, and tables are an integral part of everyday communication (e.g. Lemke 1998) (Kessler 2022: 552)

# data[].attributes.body
o   if the multimodal activity does not promote a real and authentic need to engage with the L2

--> it may actually detract from or stifle L2 development (Kessler 2022: 553)

# data[].attributes.headline
1. definition multimodality in education

# data[].attributes.subject
1. definition multimodality in education

# data[].attributes.body
Multimodality in education adopts an expanded view of literacy to include the range of multimodal communicative practices [...]. Multimodal pedagogies refer to the ways in which the teacher can design learning experiences using a range of multimodal resources (Bezemer & Kress, 2016). (Lim, Toh & Nguyen 2022:1)

# data[].attributes.headline
4. engaging with the students’ lifeworld (Lim, Toh & Nguyen 2022:4)

# data[].attributes.subject
4. engaging with the students’ lifeworld (Lim, Toh & Nguyen 2022:4)

# data[].attributes.body
o   Digital games

o   Digital books

o   Videos

o   Comics

o   Digital photo stories

o   websites

o   advertisements

--> student-centered learning

# data[].attributes.body
involves guiding the students in interpreting and evaluating the semiotic choices made in the multimodal texts, often with the support of a metalanguage that provides a vocabulary to describe the features in the texts (Lim, Cope & Kalantzis, 2022; Shin, Cimasko & Yi, 2020). (Lim, Toh & Nguyen 2022:6)

# data[].attributes.body
refer to teaching and learning activities that are process-orientated, human-centred, and are grounded on an enquiry-discovery approach (Arnott & Yelland, 2020). (Lim, Toh & Nguyen 2022:6)

# data[].attributes.headline
8.3 A culturally responsive multimodal pedagogy 

# data[].attributes.subject
8.3 A culturally responsive multimodal pedagogy 

# data[].attributes.body
includes the diverse and lived experience of the students (Kiss & Weninger, 2017) and the teacher designs the curriculum to connect home and school experiences by drawing upon students’ everyday literacy practices to develop school literacy practices. (Lim, Toh & Nguyen 2022:&)

# data[].attributes.body
o  focuses on how multimedia can facilitate learning

o combines devices (e.g., PowerPoint and lecturer's voice).

o uses words and pictures (e.g., on-screen text and animation).

o engages visual and auditory senses (e.g., slides and narration).

o Teachers role: acts as a cognitive guide to support the learner’s sense-making process. (Farias 2011: 136f.)

# data[].attributes.body
To support empirically this multimedia cognitive theory, this author has developed seven multimedia principles (Farias 2011: 136f.)

# data[].attributes.body
o  Two Channels: Brains handle words and images separately.

o  Different Processing: Words processed sequentially, images simultaneously.

o  Logogens and Imagens: Brain units for words (logogens) and images (images).

o  Learning Insights: Guides better multimedia learning design.

o  Teaching Challenge: Balance multimedia presentations to avoid cognitive overload. (Farias 2011: 138)

# data[].attributes.headline
Schnotz, Bannert & Seufert’s Model (2002)

# data[].attributes.subject
Schnotz, Bannert & Seufert’s Model (2002)

# data[].attributes.body
o  addresses reading comprehension from multimodal texts, involving both text and pictures.

o   describes how textual and visual information is represented and understood.

o  understanding both text and pictures involves creating mental representations using different sign systems and principles that complement each other. (Farias 2011: 139)

# data[].attributes.body
o  On the left side, it shows how written text is represented, with three levels:

external text, internal representation of text structure, and propositional representation of its meaning. This involves symbol processing.

o  On the right side, it shows how pictures are represented

including external picture, internal visual perception, and mental model of the content. (Farias 2011: 139)

# data[].attributes.headline
Schnotz and Bannert's Integrated Model Summary 

# data[].attributes.subject
Schnotz and Bannert's Integrated Model Summary 

# data[].attributes.body
o 1. Descriptive Representations: Representations using symbols (e.g., texts, mathematical expressions) that have no resemblance to their referents (e.g., the word "dog").

o 2. Depictive Representations: Representations using icons (e.g., photographs, drawings) that resemble their referents (e.g., a picture of a dog). effective for conveying information that resembles object or events

o Both channels limited by how much information our working memory can handle at once (Farias 2001: 139)

# data[].attributes.body
Logogen: "The mountains loomed large"

# data[].attributes.body
o Situation: Ellie and her mother in the kitchen.

o Ellie pushes away a cup of milk, indicating refusal.

o Mother interprets and verbalizes Ellie's action ("no, okay"), treating it as part of the dialogue.

o Even unintended actions by children can be integrated into communication by adults.

o Multimodal Communication: Analyzing both actions and gestures in their interactive contexts is essential for understanding how adults and children create and share meaning.

# data[].attributes.body
The father and the daughter are having breakfast. FATHER: Are you tired?
Ella whimpers and rubs her face.
FATHER: Oh a little bit.

The child then makes a very unexpected gesture. Her hand goes down along her hair. She hits her head and looks at her father as she produces a short vocalization.

ELLA: eh!
The father repeats his daughter's gesture and points to her head. FATHER: baby's head.
The father then points at his own head.
FATHER: daddy's head.