Shiraz Univ. From CMT to multimodal metaphor

lbormir@idm.upv.esGALE Research Group, Applied Linguistics DepartmentSchool of Fine ArtsPolytechnic University of Valencia (Spain)

Dr. Lorena Bort-Mir

From Conceptual Metaphor Theory to Multimodal Metaphor

04

03

Metaphor and film

Filmic metaphoricity

in different materials

02

01

Introduction

Metaphor and me

Metaphor Identification Procedures

and its critiques

Conceptual Metaphor Theory

Table of contents

07

A question for the audience

06

05

Metaphor and multimodality

Multimodal metaphor

Food for thought

Conclusions

Table of contents

1. INTRODUCTION

What do I do?

• 2022 Master's Degree "University Expert of Higher Education Pedagogy"
• 2021 Lecturer of English for Specific Purposes at the Polytechnic University of Valencia (Spain) (Computer Science School, and Faculty of Fine Arts)
• 2019 Ph.D. "Developing, Applying and Testing FILMIP", Jaume I University (Castellón, Spain)
• 2003-2015: teaching at other institutions and CEO of 2 start-ups
• 2003 Bacherlor's Degree: English Studies

Some relevant published papers

• Bort-Mir, L. & Ghaffaryan, S. (in prep.). Metaphorical Conceptualization of Inner Peace in Islamic and Christian Literature: A Comparative Analysis
• Bort-Mir, L., Bolognesi, M., & Ghaffaryan, S. (2020). Cross-cultural interpretation of filmic metaphors: A think-aloud experiment. Intercultural Pragmatics, 17(4), 389-416.
• Bort-Mir, L. (2021). Identifying metaphors in TV commercials with FILMIP: The filmic metaphor identification procedure. Journal of English Studies, 19, 23-46.
• Bort-Mir, L., & Bolognesi, M. (2022). Reliability in the identification of metaphors in (filmic) multimodal communication. Multimodal Communication, 11(3), 187-201.

Research interests, supervising Ph.D. students interested in:

Cross-cultural analyses

Abstract concepts

Multimodal Metaphor

Identification and analysis of multimodal metaphorical discourse

Is metaphor equally depicted, processed, and understood across cultures?

Analysis of metaphorical conceptualization of abstract concepts. Example: inner peace, God, faith, friendship,...

2. CONCEPTUAL METAPHOR THEORY (and its critiques)

Cognitive Linguistics + Metaphor Studies

Metaphors We Live By(Lakoff & Johnson)

1980

1979

Metaphor and Thought (Ortony)

Cognitive Linguistic turn in the study of Metaphor

• Metaphors are not just literary devices
• Metaphors are common ways of understanding reality expressed in our everyday language
• Collected linguistic data from diverse sources
• Ordinary language that people write and speak is full of linguistic metaphorical expressions
• CONCEPTUAL METAPHOR THEORY

(1980) (Hereafter L&J)

Lakoff & Johnson

Conceptual metaphor = cognitive mechanism that allows us to understand one thing in terms of another, that is, “a systematic set of correspondences between two domains of experience” (Kövecses, 2016: 14)

Usually the most abstract concept

Target domain

Source domain

Certain features from a concrete concept

are mapped onto

(Lakoff & Johnson, 1980)

Types of conceptual metaphors

Ontological metaphors

Orientational metaphors

THE MIND IS A MACHINE and INFLATION IS AN ENTITY

CONSCIOUS IS UP/UNCONSCIOUS IS DOWN, HEALTH AND LIFE ARE UP/SICKNESS AND DEATH ARE DOWN, and MORE IS UP/LESS IS DOWN

Structural metaphors

ARGUMENT IS WAR, TIME IS MONEY, and LOVE IS A JOURNEY

Lakoff and Johnson (1980) propose that conceptual metaphors fundamentally structure our conceptual system. These metaphors are not merely linguistic expressions fixed in a language's lexicon but are integral to human thought. Thus, metaphors are essential to conceptualizing and understanding the world around us.

Conclusion:

(1997)

Grady's critical view on CMT

01

Grady

(1997)

• Systematicity: which features are mapped from one domain to the other and which are not? how can we know?
• Directionality: from source to target but not from target to source
• Are metaphors so common as L&J implied and why?
• Is the bodily experience that motivates metaphors really universal for all metaphors?
• How do metaphors relate and interact between each other?
• Non-linguistic evidence for metaphor

(Grady, 1997: 15)

" if metaphor is a conceptual phenomenon rather than a specifically linguistic one, it stands to reason that it should be reflected in cognitive behaviors other than language"

(2007)

Pragglejaz 's critical view on CMT

- Methodology: L&J's linguistic metaphors were based on intuition, extracted mainly from dictionaries: data not coming from real speakers in natural discourse. - Direction of analysis: L&J's top-down analysis (conceptual metaphors extracted from a few linguistic examples). Improvement with a bottom-up analysis (complete corpora allow the structuring of conceptual metaphors with structured methodologies)

02

Pragglejaz Group

(2007)

3. Metaphor Identification Procedures

Procedures for the identification of linguistic metaphors

MIP + MIPVU

MIP and MIPVU

All phenomena that imply a connection of similarity between two distinct domains: similes, comparisons, analogies, etc. are proposed for metaphoricity.

MIPVU (Steen et al., 2010)

MIP (Pragglejaz Group, 2007)

4-step procedure identifies the basic and the contextual meaning of a lexical unit. Both meanings are compared and contrasted: decision towards metaphoricity

versus

Metaphors beyond verbal language

Assuming that metaphors are not only figures of speech but also tropes that configure our thought, it seems obvious that metaphors can be expressed through other means of communication rather than language.

Visual Metaphor Identification Procedure

VISMIP

Filmic Metaphor Identification Procedure(Second Talk, November 7)

FILMIP

MIP + MIPVU + VISMIP+ FILMIP

Applied

Tested

Accepted

Published and accepted

Applied to a wide variety of genres

Reliability tests with significant results

4. Filmic metaphoricity

Precursors: David Bordwell, Noël Carroll, Edward Branigan, Joseph Anderson, Torben Grodal, Ed Tan, Murray Smith.

Cognitive Approach to the study of film

Cognitive theory of film

Key assumptions

Cognitive psychology explains the different mental activities that take place from the spectators’ point of view as well as their filmic processing and understanding of cinematic materials

Starts in the 80s

Implying a great shift in the way scholars approached filmic studies. Lots of papers and books arise

1. We see what happens
2. We know what we would do (memory)
3. We simulate and feel (Mirron neurons, Gallese, 2005)
4. Our body mentally performs the action activating our motor system

Grodal, 2009

The PECMA FLOW

As Müller and Kappelhoff (2018: 91) well explain, “the film comes to life in the bodies of the viewers”.

So...

Films: diverse meanings

Abstract ideas that are depicted in films through metaphorical representations (Coëgnarts & Kravanja, 2012).E.g. love

Abstract meanings

Concrete meanings

Easily perceived and recognized by means of explicit, definite filmic elements:E.g. a dog

and

FILMIC METAPHORa metaphor that is based on the cinematographic modification of reality by the narrative (Coëgnarts & Kravanja, 2012b), and this modification of reality may be depicted in several ways, most of the times using different filmic elements such as colors, sounds, perspective, etc. for the construction of these metaphors.

Emotions in films are evoked by music (Agawu, 2014; Trainor and Schmidt, 2003)

Music triggers metaphorical understanding(Chattah, 2006)

Connotations of music mapped onto narrative(Chattah, 2006)

Being the 'commentator' of the events (Chattah, 2006)

Deep impact on the audicence(Agawu, 2014)

is one of the deriving studies from the cognitive approach

Music

How music can completely change a scene

5. Metaphor and Multimodality

Kress and van Leeuwen (1996)

• Multimodality involves using multiple modes
• Meaning is constructed through the interaction of these varied resources
• Multimodal communication is common across media types
• By examining how different modes combine, researchers can better understand how meaning is created and communicated.
• This approach highlights how various elements work together to shape complex messages and experiences.

What is Multimodality?

(Jewitt, 2009; Norris, 2013; Klug & Stöckl, 2016), etc.

A crucial question: What is a MODE?

Modes in films

Integrated approach

Semiotic resources

Articulatory modalities

Cienki and Müller (2008), Müller et al. (2009) and Forceville (2006)

Bort-Mir (in press), based on Norris (2013), Rossolatos (2014), Forceville (2006, 2007), and Valeiras-Jurado & Bernad-Mechó, (2022)

Pun (2008)

classification of COMMUNICATIVE MODES in films (Bort-Mir, in press)

The role of metaphor in multimodal materials

But... How?

Through a dynamic interaction of communicative modes = distinct modes work together to create meaning throughout the whole the filmic narrative

Abstract concepts

Expressed through figurative means of communication

(Forceville, 2007), Forceville & Uriós-Aparici (2009)

Giving rise to multimodal metaphors

Definition of multimodal metaphor

A metaphor in which the source domain is conveyed through one or more communicative modes, while the target domain is articulated through other modes

Metaphor in films

A potent tool

Coëgnarts & Kravanja (2012)

The experience of FALLING IN LOVE: camera movements spatially altering from periphery to centre, or fast spinning camera movements. MULTIMODAL MET: FALLING IN LOVE IS SPATIALLY ALTERING FROM PERIPHERY TO CENTRE

Coëgnarts & Kravanja (2012)

PASSAGE OF TIME IS MOTION IN SPACE

Bort-Mir (in prep.)

Sven, a reindeer, a best friend. Who is the man's bestfriend par excellence? dogs! FRIENDSHIP IS DOG-LIKE BEHAVIOR

Bort-Mir (1219)

LOVE IS VERTIGO SENSATION (spinning fast camera movements)

• Cognitive theory and neuroscience offer a wide range of reasonable explanations about the way the audience makes sense of the filmic medium (Bordwell, 1985, 1989; Carroll, 1987, 1996; Grodal, 1999).

• Metaphoric concepts are represented non-verbally through multimodal metaphors in films (Fahlenbrach, 2017; Forceville, 2009).

6. CONCLUSIONS

Can metaphors in media and film challenge our perceptions or lead to personal or societal change?Have you ever seen a metaphor or listened to it that changed the way you think about a particular topic?

7. Food for thought

Lorena Bort Mirlbormir@upv.es

Thank you

3.DEVELOPING FILMIP

(Cohn, 2016)

Temporal juxtaposition of units in time

Sense of temporality because of motion

Moving camera in film (panning, zooming)

Moving content in film

Spatial juxtaposition of units (in page layout)

No temporality of units

Static depictions in images

Static content in images

VS

Filmic narratives

Static narratives

Page 115

STEP 1

ESTABLISHING A GENERAL UNDERSTANDING

Calendario

1.4 Identifying the topic

1.3 Reconstructing the message

1.2 Description of more abstract meaning

1.1 Description of referential meaning

Lorem ipsum dolor sit

STEP 2

STRUCTURING THE REFERENTIAL DESCRIPTION

Calendario

Structured Annotation tool (Tam & Leung, 2001)

Lorem ipsum dolor sit

STEP 3

FINDING INCONGRUOUS FILMIC COMPONENTS

Calendario

Incongruity

Lorem ipsum dolor sit

STEP 4

INCONGRUITY BY COMPARISON

Calendario

What and how to compare?

Lorem ipsum dolor sit

STEP 5

TESTING CROSS-DOMAIN COMPARISON

Calendario

How to test that two concepts belong to 2 different domains?

Lorem ipsum dolor sit

STEP 6

CHECKING COMPARISON INDIRECTNESS

Calendario

Is the comparison indirect discourse?

Lorem ipsum dolor sit

STEP 7

FINAL DECISION

Calendario

If 4, 5 and 6 are positive, then metaphor.

Lorem ipsum dolor sit

4.APPLYING FILMIP

Materials, analytical instruments, and procedure

7.1 Method

MATERIALS: Corpus of 5 TV commercials

JUSTIFICATION OF THE CORPUS- (p.99) Filmic narrative expressed: atmospheric advertisement (Kenzo Homme) to avoid individual interpretations (O'Shaugnessy & O'Shaugnessy, 2003) VS. structured narrative (Agua Fresca de Rosas, Davidoff Adventure, Black Opium)- (p. 165-166) Types of metaphorical representation: cultural metaphor, multiple metaphors, no metaphor, metaphor triggered by the name of the perfume = how different metaphors are construed cross-modally- (p. 101-104) Metaphorical representation of smell

ATLAS.TI

Analytical instruments

Qualitative data analysis software - video segmenting- classification of communicative modes- networks on different variables.

SEE

SEE

SEE

Sequences, scenes and shots.

NETWORKS

ANNOTATION

CODING

SEGMENTATION

TRANSCRIPTION

CORPUS

Creation of networks with Atlas.ti

Coding segments

Coding scheme design. Codes to Atlas.ti

Dialogues, voices over and off, lyrics: Word files uploaded onto Atlas.ti

Materials selection. Clips uploaded onto Atlas.ti

PROCEDURE

of 2 TV commercials

7.2 Analysis

Positive result towards metaphoricity

1. Black Opium

Black Opium (Yves Saint Laurent, 2015)

STEP 1

ESTABLISHING A GENERAL UNDERSTANDING

Calendario

1.4 Identifying the topic

1.3 Reconstructing the message

1.2 Description of more abstract meaning

1.1 Description of referential meaning

Lorem ipsum dolor sit

Step 7: Final decision towards metaphoricity

Step 6: Checking comparison indirectness

Step 5: Testing cross-domain comparison

Step 4: Incongruity by comparison

Step 3: Finding incongruous filmic components

Step 2: Structuring the referential description

Negative result towards metaphoricity

2. Davidoff Adventure

Davidoff Adventure (Zino Davidoff, 2007)

STEP 1

ESTABLISHING A GENERAL UNDERSTANDING

Calendario

1.4 Identifying the topic

1.3 Reconstructing the message

1.2 Description of more abstract meaning

1.1 Description of referential meaning

Lorem ipsum dolor sit

Step 7: Final decision towards metaphoricity

Step 6: Checking comparison indirectness

Step 5: Testing cross-domain comparison

Step 4: Incongruity by comparison

Step 3: Finding incongruous filmic components

Step 2: Structuring the referential description

80%

RQ1: can metaphors be identified in films?RQ2: can FILMIP be applied to real filmic materials?

Where are we so far?

We have covered almost the whole project

5.TESTING FILMIP

Procedure

+ info

2 studies to check validity and reliability

+ info

+ info

+ info

Measure

What can be considered a good or bad value?

Reliability: Kappa & Alpha. Validity: percentage agreement

Reliability (Krippendorff, 2011)Validity (Cornball & Meehl, 1955)

Approaches

Definition

Reliability and validity

Study 2: Qualitative analysis

Intended to test the validity of FILMIP. Based on a percentage agreement index.

Study 1: Content analysis

Intended to test the reliability of the application of the seven steps. A content analysis based on the Kappa and Alpha reliability coefficients.

for reliability and validity tests

2 studies

21 participants watched 2 TV commercials and analyzed them with FILMIP, responding to 2 questionnaires

data collection

FILMIP'S Reliability

CLASSIFICATION OF CLAUSES UNDER CATEGORIES

CODEBOOK DESIGN WITHCATEGORIES

SEGMENTATION OF PARTICIPANTS' RESPONSES INTO CLAUSES

STUDY 1

21 responses to the 7 steps. Do they all identify the same type of units? (percentages)

STUDY 2

RESULTS!!Reliable or not?

Phase 3: Reliability test with 3 coders

Phase 2: Training session for another coder

PHASE 1:Reliability test with 2 independent coders

3 Phases for data analysis(Bolognesi et al., 2017)

Assess the content of what the participants wrote in the questionnaires and to classify that content

STUDY 1: CONTENT ANALYSIS (reliability)

RESULTS Black Opium

RESULTS Agua Fresca de Rosas

A total of 39 percentage indexes

Percentage agreement index calculated with Google Docs Graphs

Do all analysts identify the same metaphorical elements? (do they 'see' the same?)

STUDY 2: QUALITATIVE ANALYSIS (validity)

High levels of agreement among independent coders

Study 2

Study 1

• Significant percentages
• FILMIP = significant degree of validity

• Results highly significant: all above 0.7, some even reaching perfection (κ =1 α =1)
• Coding scheme = Reliable
• FILMIP = high degree of reliability

about the tests

Summary

6.GENERAL CONCLUSIONS

FILMIP: multimodal structured-semiotic analyses of filmic materials, revealing all the levels of signification and also their aesthetic configuration.

Conclusion

FILMIP is a procedure that provides consistent results among independent analysts, thus making it a valid and reliable tool for the identification of metaphorically-used components in films.

Research questionsRQ1, RQ2, RQ3

1. Can metaphors be identified in films in a consistent way?2. Can FILMIP be applied to real materials?3. Is FILMIP a reliable and valid tool?

The Filmic Metaphor Identification Procedure

6.1 CONCLUDING OVERVIEW

6.2

Limitations, implications, further research

Further research

Implications

Limitations

ApplicationCorpus

More genresLarger corporaRefinement of FILMIP: generalizability

Metaphor studies, Discourse analysis, Psychologists, Film theoryMedia studies

Branding & Marketing

Film Theory

Multimodality

Metaphor studies

FILMIP: interdisciplinarity

Thank you

Doctoral Dissertation presented by Lorena Bort MirSupervisors:Dr. Ignasi Navarro i Ferrando, Dr. Antonio José Silvestre LópezUniversitat Jaume I

The Filmic Metaphor Identification Procedure

Developing, Applying, and Testing FILMIP

External referees: Dr. Janina Wildfeuer (University of Bremen) + Dr. Aletta Dorst (University of Leiden)

"And thanks; and ever thanks"William Shakespeare, Twelfth Night (1.3)

Acknowledgements

Dr. Marianna Bolognesi (University of Oxford)

Dr. Ignasi Navarro i FerrandoDr. José Antonio Silvestre López

Mention for International Doctorate

HusbandKidsMom & sister

My family

Balaguer Gonell Hnos. Foundation

My PhD supervisors

My stay supervisor

INFO

INFO

INFO

Prospects for FILMIP

Red Hen Lab

Social community

Online public database

Website + Workshops

Atracting interest

Facebook Group

Why the Structured Annotation Tool? (Tam and Leung, 2001)Two advantages (Šorm and Steen 2018: 66): 1. limited number of components (higher inter-analyst agreement)2. the components are common to many languages (the analyst will not need specific linguistic expertise to structure their image descriptionsFILMIP assumes this procedure as valid for moving pictures: there is a need to simplify the content

Why should researchers watch the clip 5 consecutive times?"preparatory viewing" (Rossolatos, 2014: 56)Idea supported by experiment (Bort-Mir, Ghaffaryan & Bolognesi, in prep)Mental model of filmic metaphor processing:1. Participants watched TV commercial 3 times2. Participants watched TV commercial 5 times3. While verbalizing their thoughts: "the more times I saw it the more meaning I could get"

Is the use of Wordnet necessary?One of the main aims of FILMIP (+ the other structured methods) is to offer analytical guidelines so that similar conclusions can be drawn from the results of the analyses performed by different analysts. Nothing should be left for personal intuition: higher levels of inter-rater indexes.

MAIN REFERENCES- Metaphor: Lakoff & Johnson, 1980; Grady, 1997, Semino, 2002; Deignan 2005; Pragglejazz Group, 2007; Steen et al., 2010; Sorm & Steen, 2018.- Film: Bordwell, 1985, 1989, 2012; Grodal, 1999, 2009; Gallese, 2005.- Multimodality: Wildfeuer, 2012, 2014; Fahlenbrach, 2005, 2010, 2014, 2017; Forceville, 1996, 1999, 2002, 2006, 2007, 2008, 2012, 2015, 2009.