Core reading 1Core reading 1

� Multimodality:

Discipline or approach?

2

� This contribution discusses multimodal approaches to the study of linguistics, and of representation and communication more generally.

� It draws attention to the range of different modes that people use to make meaning beyond language that people use to make meaning beyond language – such as speech, gesture, gaze, image and writing – and in doing so, offers new ways of analysing language.

3

Multimodality:

� In this study, it refers to a field of applicationrather than a theory.

� Multimodality and social linguistics

� Multimodlaity and social semiotics

4

� Outline:

◦ Social linguistic approaches to multimodality

◦ Social semiotic approach to multimodality

◦ Analytical framework for analysing multimodal◦ Analytical framework for analysing multimodal

representation and communication

◦ Speech in a multimodal world

◦ Writing in a multimodal world

5

� Speech and writing are the central modes of representation and communication in a range of interrelated research traditions concerned with the social and situated use of language.

� These traditions include:� These traditions include:◦ Conversation Analysis

◦ Interactional sociology

◦ Interactional sociolinguistics

◦ Linguistic anthropology

◦ Linguistic ethnography

6

� Multimodality is differently construed in social linguistic work.

� Some studies are based on the assumption that speech or writing is always dominant, that speech or writing is always dominant, carrying the ‘essence’ of meanings, and that other simultaneously operating modes can merely expand, exemplify or modify these meanings.

7

� Different notions:

◦ Paralinguistic resources

◦ Nonverbal resources (gaze, facial expressions, gestures, etc..)gestures, etc..)

◦ ‘Contextualistion cues’ (Gumperz 1999)

� Modes analysed in conjunction with speechand writing

8

� While modes of communication other than language are, to varying degrees, being attended to in social linguistic work, its central units of analysis are usually linguistic units (e.g. ‘intonation unit’) units (e.g. ‘intonation unit’)

� In a social semiotic approach, ‘mode’ is privileged as an organizing principle of representation and communication, and therefore treated as a central unit of analysis.

9

� The starting-point for social semiotic approaches to multimodality is to extend the social interpretation of language and its meanings to the whole range of modes of representation and communication employed representation and communication employed in a culture (Kress, 2009; van Leeuwen, 2005).

� Central to this approach are three theoretical

assumptions.

10

� First, social semiotics assumes that representation and communication always draw on a multiplicity of modes, all of which contribute to meaning.

� It focuses on analysing and describing the full It focuses on analysing and describing the full repertoire of meaning-making resources which people use in different contexts (actional, visual, spoken, gestural, written, three-dimensional, and others, depending on the domain of representation), and on developing means that show how these are organized to make meaning.

11

� Second, multimodality assumes that all forms of communication (modes) have, like language, been shaped through their cultural, historical and social uses to realize social functions.functions.

� For instance, the spatial extent of a gesture, the intonational range of voice, and the direction and length of a gaze are all part of the resources for making meaning.

12

� Third, the meanings realized by any mode are always interwoven with the meanings made with those other modes co-present and co-operating in the communicative event. This interaction produces meaning. interaction produces meaning.

� Multimodality focuses on people’s process of meaning making, a process in which people make choices from a network of alternatives: selecting one modal resource (meaning potential) over another (Halliday, 1978).

� Social semiotics assumes that resources are socially shaped to become, over time, meaning-making resources which articulate the (social, individual/affective) meanings demanded by the requirements of different

14

demanded by the requirements of different communities.

� These organized sets of semiotic resources for making meaning (with) are referred to asmodes.

� For example, the way in which gesture has been shaped into modes varies across diverse communities such as the hearing impaired, ballet dancers, deep-water divers and airport runway ground staff.

15

runway ground staff.

� In order for something to ‘be a mode’ there needs to be a shared cultural sense of a set of resources within a community and how these can be organized to realize meaning.

� Modes can also be understood in terms of Halliday’s (1978) classification of meaning.

� He suggests that every sign simultaneously tells us something about ‘the world’ (ideational meaning).

16

(ideational meaning).

� Every sign positions us in relation to someone or something (interpersonal meaning).

� Every sign produces a structured text (textual meaning).

� Modal affordance, originating in the work of Gibson (1977), is a concept describing what is possible to express and represent easily in a mode. For Gibson, affordance is a matter of the material perception of the physical

17

of the material perception of the physical world.

� By contrast, social semiotics approaches affordance in relation to the material and the cultural, social-historical use of a mode. Compare speech and image, for instance.

� Sound, the material basis of speech, unfolds in time; it is sequenced. This logic of sequence in time is unavoidable in speech: one sound has to be uttered after another, one word after another, one syntactic and textual element after another.

18

textual element after another.� Marks on a surface constitute a material basis

of image, which does not unfold in time to its audience; the reader of an image can access the spatially organized constituents of the image simultaneously.

� Step 1: collecting and logging data

The log is a synopsis of what was going on during the observations

� Step 2: viewing data includes vision only, sound only, fast forward, in slow motion – all sound only, fast forward, in slow motion – all of which provide different ways of seeing the data.

� Step 3: sampling data

� Step 4: transcribing and analysing data

19

� How is English produced? What does English become when it is interactively constructed, for instance, in classrooms marked by social, cultural and linguistic diversity?

� Three analytical stages for a multimodal analysis of speech of speech

� The first starting-point foregrounds the idea of mode. A number of modes are key to the multimodal interaction: gesture, gaze, body posture, movement, spatiality, talk, writing/diagram, the teacher’s interaction with the textbook.

20

� Analysing modes individually, while analytically useful, is also problematic in the way it breaks up the interaction into separate modes. This problem is overcome by looking at all the modes together and asking how it is that they interact (this is the second analytical that they interact (this is the second analytical stage).

� The third analytical starting-point seeks to understand the communication practices of the speaker through the social principles at work across modes.

21

22

23

� What exactly has changed in the graphic designs of learning resources over the last century?

� What may have been gained and what may have been lost in potential for learning as a have been lost in potential for learning as a result of these changes?

� Effects on:◦ Segmentation

◦ Layout

◦ Directionality

24

� Graphically, segmentation implies a distinctiveuse of modes.◦ Titles

◦ Headings

◦ Margins

◦ Captions,

25

◦ Captions,

◦ Colour

◦ Font, etc.

� Bits of texts (1934): an introduction, a poem, a painting

� Blocks of texts (2002): writing and photographs

� Layout as a mode of representation:

◦ 1934 > divided blocks

◦ 2002> overlapped blocks

26

◦ 2002> overlapped blocks

� Directionality :◦ 1934> text is entirely sequentially organized, the learner

was supposed to read from page one through to the last page, and to read, on each page, from the top left corner to the bottom right corner. The order in which learners engaged with the parts of the text was fixed, by the designer.◦ 2002> the reader is required to follow more of a ‘back-

27

◦ 2002> the reader is required to follow more of a ‘back-and-forth’ principle, moving between the text blocks and the images, but leaving room for learners to pursue their preferred navigation path.

� The shift from a first-then to a back-and-forth directionality, which makes the learners’ navigation path less fixed, can be related to a shift within society from ‘vertical’ to ‘horizontal’ structures, from hierarchical to more open, participatoryrelations.

� Like any analysis of representation and communication, multimodal analysis is limited in its scope and scale.

� In terms of scope, too much attention to many different modes may take away from understanding the workings of a particular mode; too much attention to a single mode and one runs the risk of ‘tying things down’ to just one of many ways in which people make meaning.

28

ways in which people make meaning.

� As for potentials and limitations in scale, multimodal analysis is focused on micro-interaction, and therefore questions of how the analysis can speak to ‘larger’ questions about culture and society are often raised.

� This can be overcome, in part at least, by linking multimodal analysis with broader social theory and by taking into account historical contexts.