Demo

Recommender systems: exploring the effects of interpretable explainability on popular performance metrics

Saffron Zainchkovskaya

Dr Sunčica Hadžidedić

Literature

Issues:

1. Single domain exploration [3,5,6,7,8,11]
2. Lack of performance metrics available - mostly user study based
1. (not as conclusive)
3. Lack of the effect of addition of explainability on common metrics like RMSE, precision, and recall.
1. Some papers ignore standard performance metrics
4. Severe lack of human-interpretable explanations
1. Knowledge graphs
2. Cluster graphs
3. Latent factors

aims for this project

1. Understand Explainability in RSs Across Different Domains
1. Identifying any domain-specific challenges
2. Examine the effects of integrating explainability features on standard performance metrics, across multiple domains.
3. Address the lack of knowledge surrounding the effect of differing explainability incorporation on a multitude of evaluation metrics.
4. Transforming common explanability types to interpretable versions
1. Use metrics to evaluate such explainations.

OLd DELIVERABLES

ADVANCED

INTERMEDIATE

BASIC

1. Launch a complete mobile application for the RSs in the high impact domain.
2. Execute a comprehensive user studyxplainability and trust in the RS.

1. Complete data collection and cleaning for the high impact domain
2. Expand the RS & ERS to a front end application
3. Implement the back end for the high impact domain (RS*& ERS*).
4. Perform offline comparisons to assess the impact of domain on explainability and trust.

1. Complete data collection and cleaning for the low impact domain
2. Develop a baseline recommendation system (RS) and a explinable RS (ERS) in the low impact domain.
3. Conduct offline tests to compare the explainable and trustworthy RS against the baseline

NEW DELIVERABLES

ADVANCED

INTERMEDIATE

BASIC

1. Use GPT-3 API to process explanations produced by models to a more interpretable format.
2. Test this novel approach by executing a user focus group on the types of explainability produced.

1. Complete data collection and cleaning for the high-impact domain
2. Implement the back end for the high-impact domain (RS*& ERS*).
3. Perform offline comparisons to assess the impact of the domain on explainability and performance metrics.

1. Research the optimal techniques for explainability and evaluation
2. Complete data collection and cleaning for the low-impact domain
3. Develop a baseline recommendation system (RS) and an explainable RS (ERS) in the low-impact domain.
4. Conduct offline tests to compare the explainable and trustworthy RS against the baseline

Progress ON DELIVERABLES

ADVANCED

INTERMEDIATE

BASIC

1. Use ChatGPT API to process explanations produced by models to a more user-friendly format
2. Test this novel approach by executing a user focus group on the types of explainability produced.

1. Complete data collection and cleaning for the high-impact domain
2. Implement the back end for the high-impact domain (RS*& ERS*).
3. Perform offline comparisons to assess the impact of the domain on explainability and performance metrics.

1. Research the optimal techniques for explainability and evaluation
2. Complete data collection and cleaning for the low-impact domain, including text pre-processing using transformers (BERT)
3. Develop a baseline recommendation system (RS) and an explainable RS (ERS) in the low-impact domain.
4. Conduct offline tests to compare the explainable and trustworthy RS against the baseline

gpt 3 explanation with latent factors

Milestones achieved pt 1

1. Thorough research of explainability metrics
1. Collected extensive data on thids
2. Tested GPT3 for producing explanations succesfully from latent factors
3. Planned future steps

1. Completed data collection for the low-impact domain
1. Collected IMDB data using web scraping
2. Completed data cleaning for the low-impact domain
1. Processed IMDB data to retrieve descriptions, directors, and actors.
2. Used BERT transformer to generate embeddings for descriptions.
3. Completed data analysis for the low-impact domain
1. Analysed the produced data, ready to be fed into the RS

Project Plan Overview

Comparison with Actual Progress

Adjustments Made

1. Focused on back end before front end
2. Revised the gantt chart for more appropriate completion
3. Revised the time spent on data collection and analysis
4. Revised front end
5. Proposed new user study format
6. Proposed the use of a novel technique of using ChatGPT API to process the recommendations.

Challenges and Solutions

• Challenge: Data Accessibility Issues
• accessing a robust dataset for the high-impact domain.
• The initial attempt to scrape data from Glassdoor was met with technical roadblocks
• IP address getting blocked repeatedly.
• Solution: Innovative Data Collection Strategies
• Researched and found external application
• designed for dynamic web scraping, which helped bypass the IP blocking issue.

• Thought of strategy to periodically change our IP addresses
• allowing to continue data collection without interruptions. (currently working on)
• As a backup, a base dataset/ synthetic data.

Next Steps

1. Research Latent Factors method
2. Complete LF for ML
3. Create Explanations for low impact
4. Evaluate on metrics
5. Repeat for high impact

THANK YOU!

Recommender systems: exploring the effects of explainability on user trust

SAFFRON ZAINCHKOVSKAYA