Shiraz Univ. From CMT to multimodal metaphor

From Conceptual Metaphor Theory to Multimodal Metaphor

Dr. Lorena Bort-Mir

lbormir@idm.upv.esGALE Research Group, Applied Linguistics Department School of Fine Arts Polytechnic University of Valencia (Spain)

Table of contents

Introduction

01

Metaphor and me

Conceptual Metaphor Theory

02

and its critiques

Metaphor Identification Procedures

03

in different materials

Filmic metaphoricity

04

Metaphor and film

Table of contents

Metaphor and multimodality

05

Multimodal metaphor

06

Conclusions

Food for thought

07

A question for the audience

1. INTRODUCTION

What do I do?

• 2022 Master's Degree "University Expert of Higher Education Pedagogy"
• 2021 Lecturer of English for Specific Purposes at the Polytechnic University of Valencia (Spain) (Computer Science School, and Faculty of Fine Arts)
• 2019 Ph.D. "Developing, Applying and Testing FILMIP", Jaume I University (Castellón, Spain)
• 2003-2015: teaching at other institutions and CEO of 2 start-ups
• 2003 Bacherlor's Degree: English Studies

Some relevant published papers

• Bort-Mir, L. & Ghaffaryan, S. (in prep.). Metaphorical Conceptualization of Inner Peace in Islamic and Christian Literature: A Comparative Analysis
• Bort-Mir, L., Bolognesi, M., & Ghaffaryan, S. (2020). Cross-cultural interpretation of filmic metaphors: A think-aloud experiment. Intercultural Pragmatics, 17(4), 389-416.
• Bort-Mir, L. (2021). Identifying metaphors in TV commercials with FILMIP: The filmic metaphor identification procedure. Journal of English Studies, 19, 23-46.
• Bort-Mir, L., & Bolognesi, M. (2022). Reliability in the identification of metaphors in (filmic) multimodal communication. Multimodal Communication, 11(3), 187-201.

Research interests, supervising Ph.D. students interested in:

Multimodal Metaphor

Abstract concepts

Cross-cultural analyses

Analysis of metaphorical conceptualization of abstract concepts. Example: inner peace, God, faith, friendship,...

Identification and analysis of multimodal metaphorical discourse

Is metaphor equally depicted, processed, and understood across cultures?

2. CONCEPTUAL METAPHOR THEORY (and its critiques)

Cognitive Linguistics + Metaphor Studies

Cognitive Linguistic turn in the study of Metaphor

1979

1980

Metaphor and Thought (Ortony)

Metaphors We Live By(Lakoff & Johnson)

Lakoff & Johnson

(1980) (Hereafter L&J)

• Metaphors are not just literary devices
• Metaphors are common ways of understanding reality expressed in our everyday language
• Collected linguistic data from diverse sources
• Ordinary language that people write and speak is full of linguistic metaphorical expressions
• CONCEPTUAL METAPHOR THEORY

Conceptual metaphor = cognitive mechanism that allows us to understand one thing in terms of another, that is, “a systematic set of correspondences between two domains of experience” (Kövecses, 2016: 14)

Source domain

Target domain

Certain features from a concrete concept

are mapped onto

Usually the most abstract concept

Types of conceptual metaphors

(Lakoff & Johnson, 1980)

Structural metaphors

ARGUMENT IS WAR, TIME IS MONEY, and LOVE IS A JOURNEY

Orientational metaphors

CONSCIOUS IS UP/UNCONSCIOUS IS DOWN, HEALTH AND LIFE ARE UP/SICKNESS AND DEATH ARE DOWN, and MORE IS UP/LESS IS DOWN

Ontological metaphors

THE MIND IS A MACHINE and INFLATION IS AN ENTITY

Conclusion:

Lakoff and Johnson (1980) propose that conceptual metaphors fundamentally structure our conceptual system. These metaphors are not merely linguistic expressions fixed in a language's lexicon but are integral to human thought. Thus, metaphors are essential to conceptualizing and understanding the world around us.

Grady's critical view on CMT

(1997)

01

Grady

(1997)

• Systematicity: which features are mapped from one domain to the other and which are not? how can we know?
• Directionality: from source to target but not from target to source
• Are metaphors so common as L&J implied and why?
• Is the bodily experience that motivates metaphors really universal for all metaphors?
• How do metaphors relate and interact between each other?
• Non-linguistic evidence for metaphor

" if metaphor is a conceptual phenomenon rather than a specifically linguistic one, it stands to reason that it should be reflected in cognitive behaviors other than language"

(Grady, 1997: 15)

Pragglejaz 's critical view on CMT

(2007)

02

Pragglejaz Group

(2007)

- Methodology: L&J's linguistic metaphors were based on intuition, extracted mainly from dictionaries: data not coming from real speakers in natural discourse. - Direction of analysis: L&J's top-down analysis (conceptual metaphors extracted from a few linguistic examples). Improvement with a bottom-up analysis (complete corpora allow the structuring of conceptual metaphors with structured methodologies)

3. Metaphor Identification Procedures

MIP + MIPVU

Procedures for the identification of linguistic metaphors

MIP and MIPVU

MIPVU (Steen et al., 2010)

MIP (Pragglejaz Group, 2007)

All phenomena that imply a connection of similarity between two distinct domains: similes, comparisons, analogies, etc. are proposed for metaphoricity.

4-step procedure identifies the basic and the contextual meaning of a lexical unit. Both meanings are compared and contrasted: decision towards metaphoricity

versus

Metaphors beyond verbal language

Assuming that metaphors are not only figures of speech but also tropes that configure our thought, it seems obvious that metaphors can be expressed through other means of communication rather than language.

VISMIP

Visual Metaphor Identification Procedure

FILMIP

Filmic Metaphor Identification Procedure(Second Talk, November 7)

MIP + MIPVU + VISMIP+ FILMIP

Accepted

Applied

Tested

Applied to a wide variety of genres

Reliability tests with significant results

Published and accepted

4. Filmic metaphoricity

Cognitive Approach to the study of film

Precursors: David Bordwell, Noël Carroll, Edward Branigan, Joseph Anderson, Torben Grodal, Ed Tan, Murray Smith.

Cognitive theory of film

Starts in the 80s

Implying a great shift in the way scholars approached filmic studies. Lots of papers and books arise

Key assumptions

Cognitive psychology explains the different mental activities that take place from the spectators’ point of view as well as their filmic processing and understanding of cinematic materials

The PECMA FLOW

1. We see what happens
2. We know what we would do (memory)
3. We simulate and feel (Mirron neurons, Gallese, 2005)
4. Our body mentally performs the action activating our motor system

Grodal, 2009

So...

As Müller and Kappelhoff (2018: 91) well explain, “the film comes to life in the bodies of the viewers”.

Films: diverse meanings

Concrete meanings

Abstract meanings

Abstract ideas that are depicted in films through metaphorical representations (Coëgnarts & Kravanja, 2012).E.g. love

Easily perceived and recognized by means of explicit, definite filmic elements:E.g. a dog

and

FILMIC METAPHORa metaphor that is based on the cinematographic modification of reality by the narrative (Coëgnarts & Kravanja, 2012b), and this modification of reality may be depicted in several ways, most of the times using different filmic elements such as colors, sounds, perspective, etc. for the construction of these metaphors.

Music

is one of the deriving studies from the cognitive approach

Emotions in films are evoked by music (Agawu, 2014; Trainor and Schmidt, 2003)

Deep impact on the audicence(Agawu, 2014)

Connotations of music mapped onto narrative(Chattah, 2006)

Music triggers metaphorical understanding(Chattah, 2006)

Being the 'commentator' of the events (Chattah, 2006)

How music can completely change a scene

5. Metaphor and Multimodality

What is Multimodality?

Kress and van Leeuwen (1996)

• Multimodality involves using multiple modes
• Meaning is constructed through the interaction of these varied resources
• Multimodal communication is common across media types
• By examining how different modes combine, researchers can better understand how meaning is created and communicated.
• This approach highlights how various elements work together to shape complex messages and experiences.

A crucial question: What is a MODE?

(Jewitt, 2009; Norris, 2013; Klug & Stöckl, 2016), etc.

Modes in films

Articulatory modalities

Integrated approach

Semiotic resources

Cienki and Müller (2008), Müller et al. (2009) and Forceville (2006)

Bort-Mir (in press), based on Norris (2013), Rossolatos (2014), Forceville (2006, 2007), and Valeiras-Jurado & Bernad-Mechó, (2022)

Pun (2008)

classification of COMMUNICATIVE MODES in films (Bort-Mir, in press)

The role of metaphor in multimodal materials

Abstract concepts

Expressed through figurative means of communication

But... How?

Through a dynamic interaction of communicative modes = distinct modes work together to create meaning throughout the whole the filmic narrative

Giving rise to multimodal metaphors

(Forceville, 2007), Forceville & Uriós-Aparici (2009)

Metaphor in films

A potent tool

Definition of multimodal metaphor

A metaphor in which the source domain is conveyed through one or more communicative modes, while the target domain is articulated through other modes

The experience of FALLING IN LOVE: camera movements spatially altering from periphery to centre, or fast spinning camera movements. MULTIMODAL MET: FALLING IN LOVE IS SPATIALLY ALTERING FROM PERIPHERY TO CENTRE

Coëgnarts & Kravanja (2012)

PASSAGE OF TIME IS MOTION IN SPACE

Coëgnarts & Kravanja (2012)

Sven, a reindeer, a best friend. Who is the man's bestfriend par excellence? dogs! FRIENDSHIP IS DOG-LIKE BEHAVIOR

Bort-Mir (in prep.)

LOVE IS VERTIGO SENSATION (spinning fast camera movements)

Bort-Mir (1219)

6. CONCLUSIONS

• Cognitive theory and neuroscience offer a wide range of reasonable explanations about the way the audience makes sense of the filmic medium (Bordwell, 1985, 1989; Carroll, 1987, 1996; Grodal, 1999).

• Metaphoric concepts are represented non-verbally through multimodal metaphors in films (Fahlenbrach, 2017; Forceville, 2009).

7. Food for thought

Can metaphors in media and film challenge our perceptions or lead to personal or societal change? Have you ever seen a metaphor or listened to it that changed the way you think about a particular topic?

Thank you

Lorena Bort Mirlbormir@upv.es

3.DEVELOPING FILMIP

VS

(Cohn, 2016)

Static narratives

Filmic narratives

Static depictions in images

Static content in images

Moving camera in film (panning, zooming)

Moving content in film

No temporality of units

Sense of temporality because of motion

Spatial juxtaposition of units (in page layout)

Temporal juxtaposition of units in time

Page 115

Calendario

ESTABLISHING A GENERAL UNDERSTANDING

STEP 1

Lorem ipsum dolor sit

1.1 Description of referential meaning

1.2 Description of more abstract meaning

1.4 Identifying the topic

1.3 Reconstructing the message

Calendario

Structured Annotation tool (Tam & Leung, 2001)

STEP 2

STRUCTURING THE REFERENTIAL DESCRIPTION

Lorem ipsum dolor sit

Calendario

Incongruity

STEP 3

FINDING INCONGRUOUS FILMIC COMPONENTS

Lorem ipsum dolor sit

Calendario

What and how to compare?

STEP 4

INCONGRUITY BY COMPARISON

Lorem ipsum dolor sit

How to test that two concepts belong to 2 different domains?

Calendario

STEP 5

TESTING CROSS-DOMAIN COMPARISON

Lorem ipsum dolor sit

Calendario

Is the comparison indirect discourse?

STEP 6

CHECKING COMPARISON INDIRECTNESS

Lorem ipsum dolor sit

Calendario

If 4, 5 and 6 are positive, then metaphor.

STEP 7

FINAL DECISION

Lorem ipsum dolor sit

4.APPLYING FILMIP

7.1 Method

Materials, analytical instruments, and procedure

MATERIALS: Corpus of 5 TV commercials

JUSTIFICATION OF THE CORPUS - (p.99) Filmic narrative expressed: atmospheric advertisement (Kenzo Homme) to avoid individual interpretations (O'Shaugnessy & O'Shaugnessy, 2003) VS. structured narrative (Agua Fresca de Rosas, Davidoff Adventure, Black Opium) - (p. 165-166) Types of metaphorical representation: cultural metaphor, multiple metaphors, no metaphor, metaphor triggered by the name of the perfume = how different metaphors are construed cross-modally - (p. 101-104) Metaphorical representation of smell

Analytical instruments

ATLAS.TI

Qualitative data analysis software - video segmenting - classification of communicative modes - networks on different variables.

PROCEDURE

SEGMENTATION

CODING

CORPUS

TRANSCRIPTION

NETWORKS

ANNOTATION

Materials selection. Clips uploaded onto Atlas.ti

Coding segments

Creation of networks with Atlas.ti

Dialogues, voices over and off, lyrics: Word files uploaded onto Atlas.ti

Coding scheme design. Codes to Atlas.ti

Sequences, scenes and shots.

SEE

SEE

SEE

7.2 Analysis

of 2 TV commercials

1. Black Opium

Positive result towards metaphoricity

Black Opium (Yves Saint Laurent, 2015)

Calendario

ESTABLISHING A GENERAL UNDERSTANDING

STEP 1

Lorem ipsum dolor sit

1.1 Description of referential meaning

1.2 Description of more abstract meaning

1.3 Reconstructing the message

1.4 Identifying the topic

Step 2: Structuring the referential description

Step 3: Finding incongruous filmic components

Step 4: Incongruity by comparison

Step 7: Final decision towards metaphoricity

Step 5: Testing cross-domain comparison

Step 6: Checking comparison indirectness

2. Davidoff Adventure

Negative result towards metaphoricity

Davidoff Adventure (Zino Davidoff, 2007)

Calendario

ESTABLISHING A GENERAL UNDERSTANDING

STEP 1

Lorem ipsum dolor sit

1.1 Description of referential meaning

1.2 Description of more abstract meaning

1.3 Reconstructing the message

1.4 Identifying the topic

Step 2: Structuring the referential description

Step 3: Finding incongruous filmic components

Step 4: Incongruity by comparison

Step 7: Final decision towards metaphoricity

Step 6: Checking comparison indirectness

Step 5: Testing cross-domain comparison

Where are we so far?

We have covered almost the whole project

80%

RQ1: can metaphors be identified in films?RQ2: can FILMIP be applied to real filmic materials?

5.TESTING FILMIP

Reliability and validity

Definition

Approaches

Measure

Procedure

Reliability (Krippendorff, 2011)Validity (Cornball & Meehl, 1955)

What can be considered a good or bad value?

Reliability: Kappa & Alpha. Validity: percentage agreement

2 studies to check validity and reliability

+ info

+ info

+ info

+ info

2 studies

for reliability and validity tests

Study 1: Content analysis

Intended to test the reliability of the application of the seven steps. A content analysis based on the Kappa and Alpha reliability coefficients.

Study 2: Qualitative analysis

Intended to test the validity of FILMIP. Based on a percentage agreement index.

data collection

21 participants watched 2 TV commercials and analyzed them with FILMIP, responding to 2 questionnaires

STUDY 1

SEGMENTATION OF PARTICIPANTS' RESPONSES INTO CLAUSES

FILMIP'S Reliability

CODEBOOK DESIGN WITHCATEGORIES

CLASSIFICATION OF CLAUSES UNDER CATEGORIES

STUDY 2

21 responses to the 7 steps. Do they all identify the same type of units? (percentages)

STUDY 1: CONTENT ANALYSIS (reliability)

Assess the content of what the participants wrote in the questionnaires and to classify that content

Phase 2: Training session for another coder

Phase 3: Reliability test with 3 coders

3 Phases for data analysis(Bolognesi et al., 2017)

PHASE 1:Reliability test with 2 independent coders

RESULTS!!Reliable or not?

STUDY 2: QUALITATIVE ANALYSIS (validity)

Do all analysts identify the same metaphorical elements? (do they 'see' the same?)

RESULTS Agua Fresca de Rosas

Percentage agreement index calculated with Google Docs Graphs

RESULTS Black Opium

A total of 39 percentage indexes

Summary

about the tests

Study 1

Study 2

• Significant percentages
• FILMIP = significant degree of validity

• Results highly significant: all above 0.7, some even reaching perfection (κ =1 α =1)
• Coding scheme = Reliable
• FILMIP = high degree of reliability

High levels of agreement among independent coders

6.GENERAL CONCLUSIONS

6.1 CONCLUDING OVERVIEW

The Filmic Metaphor Identification Procedure

Research questionsRQ1, RQ2, RQ3

Conclusion

1. Can metaphors be identified in films in a consistent way?2. Can FILMIP be applied to real materials? 3. Is FILMIP a reliable and valid tool?

FILMIP is a procedure that provides consistent results among independent analysts, thus making it a valid and reliable tool for the identification of metaphorically-used components in films.

FILMIP: multimodal structured-semiotic analyses of filmic materials, revealing all the levels of signification and also their aesthetic configuration.

6.2

Limitations, implications, further research

Limitations

Further research

Implications

Metaphor studies, Discourse analysis, Psychologists, Film theoryMedia studies

ApplicationCorpus

More genresLarger corpora Refinement of FILMIP: generalizability

FILMIP: interdisciplinarity

Multimodality

Film Theory

Branding & Marketing

Metaphor studies

Thank you

Developing, Applying, and Testing FILMIP

The Filmic Metaphor Identification Procedure

Doctoral Dissertation presented by Lorena Bort MirSupervisors: Dr. Ignasi Navarro i Ferrando, Dr. Antonio José Silvestre López Universitat Jaume I

Acknowledgements

"And thanks; and ever thanks"William Shakespeare, Twelfth Night (1.3)

My PhD supervisors

Dr. Ignasi Navarro i FerrandoDr. José Antonio Silvestre López

My stay supervisor

Balaguer Gonell Hnos. Foundation

My family

Dr. Marianna Bolognesi (University of Oxford)

HusbandKids Mom & sister

Mention for International Doctorate

External referees: Dr. Janina Wildfeuer (University of Bremen) + Dr. Aletta Dorst (University of Leiden)

Prospects for FILMIP

Online public database

Red Hen Lab

Social community

Website + Workshops

Atracting interest

Facebook Group

INFO

INFO

INFO

Why the Structured Annotation Tool? (Tam and Leung, 2001) Two advantages (Šorm and Steen 2018: 66): 1. limited number of components (higher inter-analyst agreement) 2. the components are common to many languages (the analyst will not need specific linguistic expertise to structure their image descriptions FILMIP assumes this procedure as valid for moving pictures: there is a need to simplify the content

Why should researchers watch the clip 5 consecutive times? "preparatory viewing" (Rossolatos, 2014: 56) Idea supported by experiment (Bort-Mir, Ghaffaryan & Bolognesi, in prep) Mental model of filmic metaphor processing: 1. Participants watched TV commercial 3 times 2. Participants watched TV commercial 5 times 3. While verbalizing their thoughts: "the more times I saw it the more meaning I could get"

Is the use of Wordnet necessary? One of the main aims of FILMIP (+ the other structured methods) is to offer analytical guidelines so that similar conclusions can be drawn from the results of the analyses performed by different analysts. Nothing should be left for personal intuition: higher levels of inter-rater indexes.

MAIN REFERENCES - Metaphor: Lakoff & Johnson, 1980; Grady, 1997, Semino, 2002; Deignan 2005; Pragglejazz Group, 2007; Steen et al., 2010; Sorm & Steen, 2018. - Film: Bordwell, 1985, 1989, 2012; Grodal, 1999, 2009; Gallese, 2005. - Multimodality: Wildfeuer, 2012, 2014; Fahlenbrach, 2005, 2010, 2014, 2017; Forceville, 1996, 1999, 2002, 2006, 2007, 2008, 2012, 2015, 2009.