The Role of Transparency in Recommender Systems

The Role of Transparency in Recommender Systems

• 1. The Role of Transparency in Recommender Systems Rashmi Sinha & Kirsten Swearingen School of Information Management & Systems, UC Berkeley Berkeley, CA 94720 USA {sinha, kirstens}@sims.berkeley.edu ABSTRACT Recommender Systems act as personalized decision guides, aiding users in decisions on matters related to personal taste. Most previous research on Recommender Systems has focused on the statistical accuracy of the algorithms driving the systems, with little emphasis on interface issues and the user’s perspective. The goal of this research was to examine the role of transparency (user understanding of why a particular recommendation was made) in Recommender Systems. To explore this issue, we conducted a user study of five music Recommender Systems. Preliminary results indicate that users like and feel more confident about recommendations that they perceive as transparent. Keywords: Recommender Systems, Usability Studies, WWW INTRODUCTION A good way to personalize recommendations for an individual is to identify people with similar interests and recommend items that have interested these like-minded people (Resnick & Varian, 1997). This premise is the statistical basis of collaborative filtering (CF) algorithms that form the backbone of most Recommender Systems. Many online commerce sites (Amazon, CDNow, Barnes & Noble) incorporate Recommender Systems to help customers choose from large catalogues of products. Most online Recommender Systems act like black boxes, not offering the user any insight into the system logic or justification for the recommendations. The typical interaction paradigm involves asking the user for some input (in the form of ratings of items, or listing favorite artists or authors), processing the input, and giving the user some output in the form of recommendations. We challenge this black box approach, especially since the human analogue to recommendation-making is a transparent social process. In fact, CF algorithms are also called social filtering algorithms because they are modeled after the timetested social process of receiving recommendations by asking friends with similar tastes to recommend movies, books, or music that they like. The recipient of a recommendation has a number of ways to decide whether to trust the recommendation: (a) scrutinizing the similarity between the taste of the recipient and the recommender (b) assessing the success of prior suggestions from this recommender, and (c) asking the recommender for more information about why the recommendation was made. Similarly, Recommender Systems need to offer the user some ways to judge the appropriateness of recommendations. TRANSPARENCY IN USER INTERFACES Previous research has shown that expert systems that act as decision guides need to provide explanations and justifications for their advice (Buchanan & Shortcliffe, 1984). Studies with search engines also show the importance of transparency. Koenmann & Belkin (1996) found that greater visibility and interactivity for relevance feedback helped search performance and satisfaction with system. Muramatsu & Pratt (2001) investigated user mental models for typical query transformations done by search engines during the retrieval process (e.g., stop word removal, term suffix expansion). They found that making such transformations transparent to users improves search performance. Johnson & Johnson (1993) point out that explanations play a crucial role in the interaction between users and complex systems. According to their research, one purpose of explanation is to illustrate the relationship between antecedent and consequent (i.e., between cause and effect). In the context of recommender systems, understanding the relationship between the input to the system (ratings made by user) and output (recommendations) allows the user to initiate a predictable and efficient interaction with the system. Transparency allows users to meaningfully revise the input in order to improve recommendations, rather than making “shots in the dark.” Herlocker et al. suggest that Recommender Systems have not been used in high-risk decision-making because of a lack of transparency (2000). While users might take a chance on an opaque movie recommendation, they might be unwilling to commit to a vacation spot without understanding the reasoning behind such a recommendation. STUDY GOALS We are interested in exploring the role of transparency in recommender systems. Do users perceive Recommender System logic to be transparent, or do they feel that they lack insight into why an item is recommended? Is perceived transparency related to greater liking for recommendations? Our investigation focuses on the perception of transparency rather than the accuracy of the perception. We hypothesize that users who perceive that they do not understand why a recommendation is made are less likely to trust system suggestions.

• 2. STUDY DESIGN To examine our hypothesis, we conducted a user study of 5 Music Recommender Systems (Amazon’s Recommendations Explorer, CDNow’s Album Advisor, MediaUnbound, MoodLogic’s Filters Browser, and SongExplorer). We focused on Music Recommender Systems because unlike book and movie systems, music systems allow users to evaluate each recommended item during the study itself. We identified more than 10 Music Recommender Systems and chose five that offered different interaction paradigms and results displays (e.g. number of items returned, amount of item description, audio sample vs. complete song, methods for generating new sets of recommendations). 12 people participated in the study. All participants reported using the Internet and/or listening to music daily; most reported purchasing or downloading music monthly. For each of the 5 systems (in random order), participants: (a) provided input (e.g. favorite musician, ratings on a set of items). (b) listened to music sample, and (c) rated liking, confidence & transparency for 10 recommendations. Thus current results are based on ratings of 600 recommendations (12 users x 5 systems x 10 recommendations). Our primary measure for transparency was the user response to the question, “Do you understand why the system recommended this item to you?” RESULTS Table 1 shows means and standard errors for transparent and non-transparent recommendations. Mean liking was significantly higher for transparent than non-transparent recommendations [t(579)=-7.89; p<.01]. Mean Confidence showed a similar trend [t(388)=-5.88; p<.01]. We found similar results when examining each of the five systems individually. Table 1: Effect of Transparency on Liking and Confidence Not Transparent Transparent Mean Liking 2.79 (.07) 3.51 (.06) Mean Confidence 6.89 (.17) 8.12 (.12) A potential confounding factor is users’ previous familiarity with recommended items. There are three conditions: (i) no previous experience with recommended item, (ii) a positive previous opinion of recommendation, or (iii) a negative previous experience of item. To examine the question of whether transparency and previous experience interact, we computed means for transparent and non-transparent recommendations for the three kinds of familiarity separately. Figure 1 shows that mean liking is highest for previously liked items, lowest for previously disliked items, and in between for new items. Also, the relationship between transparency and liking is modulated by a user’s previous experience with an item. Mean liking is significantly higher for transparent than non-transparent recommendations in the case of both new [t(39)=-3.7; p<.01] and previously liked [t(240)=2.68; p<.01] recommendations. The same difference was not significant for previously disliked items [t(129)=-1.08; n.s.]. Figure 1: Relationship of Transparency & Familiarity with Liking 1 2 3 4 5 New Negative Previous Opinion Positive Previous Opinion Mean Liking Not Transparent Transparent Error Bars show Std. Error DISCUSSION The results of this preliminary study indicate that in general users like and feel more confident in recommendations perceived as transparent. For new items, it is not surprising that users like transparent recommendations more than nontransparent ones. However, users also like to know why an item was recommended even for items they already liked. This suggests that users are not just looking for blind recommendations from a system, but are also looking for a justification of the system’s choice. This is an important finding from the perspective of system designers. A good CF algorithm that generates accurate recommendations is not enough to constitute a useful system from the users’ perspective. The system needs to convey to the user its inner logic and why a particular recommendation is suitable for them. We plan to follow up this work with an experimental study manipulating the transparency of recommended items while holding everything else constant. We also plan to investigate the efficacy of different methods for achieving system transparency. REFERENCES 1. Buchanan, B., & Shortcliffe, E. Rule-Based Expert Systems: The Mycin Experiments of the Stanford Heuristic Programming Project. Reading, MA: Addison Wesley Publishing Company. 1984 2. Herlocker, J., Konstan, J.A., & Riedl, J. Explaining Collaborative Filtering Recommendations. ACM 2000 Conference on CSCW. 2000 3. Johnson, J. & Johnson, P. Explanation facilities and interactive systems. In Proceedings of Intelligent User Interfaces '93. (159-166). 1993 4. Koenemann, J., & Belkin, N. A case for interaction: A study of interactive information retrieval behavior and effectiveness. In Proceedings of the Human Factors in Computing Systems Conference. ACM Press, NY, 1996 5. Muramatsu, J., & Pratt, W. Transparent Search Queries: Investigating users’ mental models of search engines. In Proceedings of SIGIR Conference. ACM Press, NY, 2001 6. Resnick, P, and Varian, H.R. Recommender Systems. Commun. ACM 40, 3 (56-58). 1997 View publication stats