VidPP

Dr Sunčica Hadžidedić

Saffron Zainchkovskaya

Recommender systems: exploring the effects of explainability on user trust

Motivation

Planning

Deliverables

Issues

Team

Approach

PROJECT OVERVIEW

Problem Statement and the focus of the project

Background

INTRODUCTION

VIDEO OVERVIEW

[1]Hallinan, B., & Striphas, T. (2016). Recommended for you: The Netflix Prize and the production of algorithmic culture. New Media & Society, 18(1), 117-137. https://doi.org/10.1177/1461444814538646

BACKGROUND

• Recommender Systems (RSs) are algorithms that provide personalised recommendations to users.
• RSs analyse patterns and infer preferences from historical user data. Such as:
• past purchases
• viewing histories
• ratings

• Popularised partially due to the Netflix $1 million competition. [1]
• to build an algorithm that beats Netflix's baseline algorithm by 10 percent.
• Commoly used in large producs used on a daily basis:
• e-commerce ( Amazon )
• entertainment ( Spotify )
• content consumption ( TikTok )

• RS operate as "black boxes"
• making it challenging to understand how recommendations are generated.

• The lack of explainability in RSs could lower user trust and satisfaction.
• As users may feel uneasy about trusting recommendations

• RSs have far-reaching implications for people's daily lives.
• There is a growing need for greater awareness of its effects.
• RSs can be used in both high impact domains

• High impact domain: has high impact on a users' life

• Alters the trajectory of their life
• E.G. Job searching

USER TRUST

RSs HAVE HIGH IMPACT

LACK OF TRANSPARENCY

ISSUES

MOTIVATION

• Motivation behind this project
• Understand the impact of explainability on the end user on a deeper level, using multiple testing techniques
• To assess the impacts of an increased level of explainability on the levels of a user’s trust with the system.
• See whether there is a difference when the impact level of the domain is different.

• WHY
• By addressing user trust concerns we can enhance user satisfaction and system adoption
• benefitting both users and organisations.

Project probleM

• Central problem addressed by this project = the lack of knowledge surrounding the effects of the incorporation of explainability within RSs
• and the impact this has on a user’s trust level with the system

• Also addresses the problem seen commonly within literature
• where only one domain is being investigated

USERS HESITATE TO RELY ON RSs

VITAL TO BOTH USERS AND ORGANISATIONS

RSs USED OFTEN & INFLUENCE US DAILY

Project Importance

• The importance of addressing this problem lies in the growing role of RSs in our daily lives

• RSs shape user experiences and decision-making processes

• Without transparency and trust, users may hesitate to rely on RS recommendations,
• leading to missed opportunities for businesses and a less satisfying user experience.
• Addressing these issues is crucial
• Users may lose trust in the system and feel dissatisfied with the suggestions provided.

• The project is unique as it systematically evaluating the effects of explainability on user trust within both low and high impact domains

• provides a comprehensive understanding of the relationship between explainability and trust in RSs.

RESEARCH QUESTION

To what extent does the integration of explainability into a recommendation system enhance the level of user trust, spanning both high and low-impact domains?

DEVELOP 4 RS BACK END

CONDUCT OFFLINE TESTS

Multi-faceted approach

CONDUCT USER STUDIES

DEVELOP THE FRONT END

• Multi-faceted approach

• Project's methodology
• Develop 4 RSs
• 2 low impact, 1 explainable, 1 base
• 2 high impact, 1 explainable, 1 not
• conduct a series of rigorous offline tests
• conduct a number of qualitative user studies
• Evaluate the results of both tests to draw conclusions about the RQ at hand.

How It Will Be Addressed

1. Create a mobile application front end for the high impact domain, integrating the backend RS* and ETRS* to it.
2. Conduct a user study on all developed RSs on at least 20 users.
3. Compare RS1 and ETRS to RS1* and ETRS*, using the qualitative (user-study) tests to identify the influence of the domain on the incorporation of explainability and user trust in RSs.

1. Create a mobile application front end for low impact domain, integrating the backend RS and ETRS to it.
2. Implement both RSs in the high impact domain [RS1*, ETRS*].
3. Compare ETRS* to a baseline RS1*, using quantitative measurements using offline tests.
4. Compare RS1 and ETRS to RS1* and ETRS*, using the quantitative, offline tests to identify the influence of the domain on the incorporation of explainability and user trust in RSs.

1. Select appropriate algorithms and metrics for explainability and user trust.
2. Create the back end for the baseline RS (RS1) in the low impact domain.
3. Create the back end for an explainable, and trustworthy RS [ETRS] in a low impact domain.
4. Compare ETRS to a baseline RS1, using quantative measurements in an offline test.

INTERMEDIATE

ADVANCED

BASIC

DELIVERABLES

PLANNING

PLANNING

SUCCESS METRICS- PRODUCT

Evaluation Criteria:

1. Accuracy: How accurate is the RSs? Has the incorporation of explainability lowered the accuracy?
2. Explainability: Is the incorporation of explainable recommendations within systems beneficial to users?
3. User Trust: Does this incorporation enhance the trust users have with the system?
4. Domains: Is the incorporation of explainations more important in a high/low impact domain?
5. Project Completion: How effectively was the project completed within the stipulated timeframe and project scope? Does the completion status of the project reflect on the quality and reliability of the RSs?
1. The project will be evaluated on a bi-weekly basis to ensure it is consistently on track.
2. During this evaluation the project components below will be examined

SUCCESS METRICS- PROJECT

PROJECT EVALUATION

Evaluation Methodologies:The product will be evaluated using three main methodologies.

1. User Study: Conduct a series of user studies on 20 or more individuals
2. Offline Test: Test the system fully on the metrics mentioned prior.
3. Comparison: Process and compare the results of each of the studies across the differing domain to draw conclusions for the project.
1. Evaluate the effect of the proposed methodology when comparing the results

OFFLINE TEST

COMPARISON

USER STUDY

PROJECT EVALUATION

• User study performed
• 20+ participants
• Process:
• Let user interact with the RS (Observed)
• Questions are permitted but users arent guided through the process
• Users interact base and ETRS in one of the two domains (randomised order)
• Questionnaire
• Break
• Users then interact with the base and ETRS in the other domain
• Questionnaire

Project Evlauation - USER STUDY

Recommender systems: exploring the effects of explainability on user trust

SAFFRON ZAINCHKOVSKAYA

THANK YOU!

SUCCESS METRICS- PRODUCT

Evaluation Criteria:

1. Accuracy: How accurate is the RSs? Has the incorporation of explainability lowered the accuracy?
2. Explainability: Is the incorporation of explainable recommendations within systems beneficial to users?
3. User Trust: Does this incorporation enhance the trust users have with the system?
4. Domains: Is the incorporation of explainations more important in a high/low impact domain?
5. Project Completion: How effectively was the project completed within the stipulated timeframe and project scope? Does the completion status of the project reflect on the quality and reliability of the RSs?
1. The project will be evaluated on a bi-weekly basis to ensure it is consistently on track.
2. During this evaluation the project components below will be examined

SUCCESS METRICS- PROJECT

PROJECT EVALUATION