Decision Making

Do as AI say: susceptibility in deployment of clinical decision-aids

Susanne Gaube Harini Suresh, Martina Raue, Alexander Merritt, Seth J. Berkowitz, Eva Lermer Joseph F. Coughlin, John V. Guttag, Errol Colak and Marzyeh Ghassemi

Contini Sara, Marcaccio Gianmarco

Index

Introduction

1.

Methods

2.

3.

Results

4.

Conclusions

1. INTRODUCTION

IS IT POSSIBLE TO INTEGRATE AI WITH HUMAN DECISION-MAKERS TO IMPROVE CLINICAL OUTCOMES?

• if the physicians are able to balance trust and skepticism
• If physicians do not trust the technology, they will not use it --> BUT blind trust can lead to medical errors

Short answer: we don't know yet

- Gaube et al., 2021

Keywords:

AI, advice, diagnosis,

2. METHODS

Radiologists vs physicians

Physicians (low expertise)

Radiologists (high expertise)

138

Sample size

127

2.1 METHODS

DESIGN

6 ACCURATE

AI ADVICE

They received 8 chest X-rays images with:

• a short clinical vignette

• diagnostic advice

2 INACCURATE

CHEST X-RAYS

DIAGNOSTIC ADVICE

6 ACCURATE

HUMAN ADVICE

2 INACCURATE

2.2 METHODS

Measures

INSTRUCTIONS

AI: “The findings and primary diagnoses were generated by CHEST-AI, a well-trained, deep-learning-based artificial intelligence (AI) model with a performance record (regarding diagnostic sensitivity and specificity) on par with experts in the field". Human: “The findings and primary diagnoses were generated by Dr. S. Johnson, an experienced riadiologist with a performance record (regarding sensitivity and specificity) on par with experts in the field"

Below the patient case, participants were asked to respond to 6 questions to measure:

1. advice quality
2. diagnosis accuracy

After all 8 cases they completed a short survey, including demographics

DIAGNOSTIC ACCURACY

2.2 METHODS

Measures

INSTRUCTIONS

AI: “The findings and primary diagnoses were generated by CHEST-AI, a well-trained, deep-learning-based artificial intelligence (AI) model with a performance record (regarding diagnostic sensitivity and specificity) on par with experts in the field". Human: “The findings and primary diagnoses were generated by Dr. S. Johnson, an experienced riadiologist with a performance record (regarding sensitivity and specificity) on par with experts in the field"

Below the patient case, participants were asked to respond to 6 questions to measure:

1. advice quality
2. diagnosis accuracy

After all 8 cases they completed a short survey, including demographics

DIAGNOSTIC ACCURACY

3.3 RESULTS

VARIABILITY IN CLINICAL PERFORMANCE

• Accurate advice significantly improves diagnostic accuracy for both radiologists and IM/EM physicians

• Physicians are influenced by advice regardless of its source (AI or human).

• Radiologists rate AI advice as lower quality but still rely on it.

IM/EM physicians didn’t show any algorithmic aversion and exhibit higher trust in AI

• Individual susceptibility leads to inaccurate advice varies widely.

Use graphics in your presentation...

3 RESULTS

ADVICE QUALITY RATINGS

Use graphics in your presentation...

3 RESULTS

INDIVIDUAL SUSCEPTIBILITY

Variability in susceptibility to inaccurate advice: - 41.73% of IM/EM physicians are highly susceptible to inaccurate advice. - 27.54% of radiologists are susceptible but maintain higher overall accuracy. - Certain participants consistently resisted inaccurate advice, showing stronger critical thinking abilities.

Use graphics in your presentation...

3 RESULTS

VARIABILITY IN CLINICAL RESPONSE

Significant variability in individual participant performance: - Some radiologists achieved perfect accuracy even with inaccurate advice. - Others struggled, especially IM/EM physicians, with 27.56% scoring ≤ 50% accuracy. - Cognitive biases, such as anchoring and confirmation bias, influence susceptibility to advice errors.

Use graphics in your presentation...

3 DISCUSSION

KEY FINDINGS

Physicians are influenced by advice regardless of its source (AI or human). Cognitive biases, like confirmation bias, lead to errors when advice is inaccurate. Radiologists rate AI advice as lower quality but still rely on it. IM/EM physicians show no aversion and exhibit higher trust in AI. Individual susceptibility to inaccurate advice varies widely. Complex cases amplify errors, highlighting diagnostic challenges.

Use graphics in your presentation...

3 DISCUSSION

KEY FINDINGS

Physicians are influenced by advice regardless of its source (AI or human). Cognitive biases, like confirmation bias, lead to errors when advice is inaccurate. Radiologists rate AI advice as lower quality but still rely on it. IM/EM physicians show no aversion and exhibit higher trust in AI. Individual susceptibility to inaccurate advice varies widely. Complex cases amplify errors, highlighting diagnostic challenges

Use graphics in your presentation...

4. Hypothesis

Synthesis and organization, the two pillars for presenting:

Accurate advice significantly improves diagnostic accuracy for both radiologists and IM/EM physicians: - Radiologists: 40.10% improvement. - IM/EM physicians: 37.53% improvement. - Inaccurate advice significantly reduces diagnostic accuracy for both groups. The source of the advice (AI vs. human) did not influence diagnostic performance.

Hypothesis 3

Hypothesis 2

Hypothesis 1

Narrative beings. We teach through stories. They entertain us and help us maintain focus.

Social beings. We need to interact with each other. We learn collaboratively.

We are visual beings. We are capable of understanding images from millions of years ago, even from other cultures.

5. Theoretical Framework

Use images in your presentation. And use this space to describe it. To insert an image, you can upload it from your computer or use the resources found within the tool.

Background

To create your interactive image... You can use a photograph or an illustration as a base. The important thing is that it is very attractive to catch the attention of your audience.

6. Methodology

Field work

That's why, at Genially, we create our designs to make understanding and learning easier, so you can level up with interactivity and turn your content into something that adds value and hooks.

Population and sample

In this space, it is also advisable to establish the importance of the topic, discuss previous and current research in the field, and identify the problem. We also recommend that interactivity and animation be present in all your creations, regardless of their type.

6. Methodology

Procedures and variables

When carrying out a presentation, one must pursue two objectives: conveying information and avoiding yawns. To achieve this, it may be a good practice to create an outline and use words and concepts that aid in the assimilation of the content.

Instruments

Interactivity and animation can make even the most boring content fun. In Genially, we also create our designs to facilitate understanding and learning, so that you level up with interactivity and turn your content into something that brings value and engages.

7. State of the issue

Interactive Visual Communication step by step:

Visual content is a cross-cutting and universal language, like music. We are able to understand images from millions of years ago, even from other cultures.

Plan the structure of your content. Give visual weight to the key points and main elements.

Define secondary messages with interactivity.

Set a flow through the content. Measure the results.

8. Development

Interactivity and animation can be your best allies in making content fun.

50%

Interactivity and animation can be your best allies when it comes to creating tables, infographics, or charts that help provide context to the topic or unit, as well as simplify information to make it more understandable. We are visual beings and find it easier to 'read' images than to read written text.

We are visual beings

8. Development

Doctor’s Orders or AI’s Suggestion? It Depends! (by ChatGPT)

'Your content is well-liked, but only captivating if it's interactive'

8. Development

Use tables and infographics

Visual communication is a key tool. We find it easier to 'read' images than to read written text. That's why disciplines like Visual Thinking facilitate knowledge organization through the use of images, graphics, infographics, and simple drawings.

8. Development

1.1. Write an awesome subtitle here

With Genially templates, you can include visual resources to engage the class from minute 1. You can also highlight key content to facilitate its assimilation and even embed external content that surprises and provides more context to the topic: videos, photos, audios... Whatever you want!

9. Conclusions

Activate and amaze your audience

Measure results and experiment

Generate experiences with your content

Show enthusiasm, smile, and maintain eye contact with your audience: 'The eyes, chico. They never lie'. This will help you make a 'match' with your audience. Leave them speechless!

You can develop the content in more detail through your oral presentation. We recommend that you train your voice and practice: the best improvisation is always the most rehearsed!

You can create an outline to synthesize the content and use words that will be etched into your audience's brains. Numbered ideas are much better remembered than bullet lists.

Discussion

10. Bibliography

Bibliographic references

• Book Title

Last Name Last Name, Author (20XX). Place of publication: Publisher.

• Book Title

Last Name Last Name, Author (20XX). Place of publication: Publisher.

• Book Title

Last Name Last Name, Author (20XX). Place of publication: Publisher.

• Book Title

Last Name Last Name, Author (20XX). Place of Publication: Publisher.

• Book Title

Last Name Last Name, Author (20XX). Place of publication: Publisher.

• Book Title

Last name Last name, Author (20XX). Place of publication: Publisher.

11. Figures and Tables

Figure 3

Figure 4

Figure 2

Figure 1

Graph 2

Table 1

Graph 1

Figure 5

Figure 8

Figure 7

Figure 6

12. Attachments

Attachments

If your thing is to capture the attention of your audience, make a 'match' with your public and leave everyone speechless, this Genially template is for you. Visual content has an incredible informative and illustrative power. That's why including videos and images in your introduction is essential.

Attachment 1

Contextualize your topic with a great subtitle here

Annex 2

Contextualize your topic with a awesome subtitle here

Annex 3

Contextualize your topic with a great subtitle here

Thank you!

Any questions?

Write a cool title here

The visual content is a transversal, universal language, like music. We are able to understand images from thousands of years ago, even from other cultures. We don't like boring. We don't want to be repetitive. Communicating in the same way is boring and doesn't engage. We do it differently. We sabotage boredom. We create what the brain likes to consume because it stimulates it.

Write a cool title here

The visual content is a transversal, universal language, like music. We are able to understand images from thousands of years ago, even from other cultures. We don't like boring. We don't want to be repetitive. Communicating in the same way is boring and doesn't engage. We do it differently. We sabotage boredom. We create what the brain likes to consume because it stimulates it.