Learning from User Interactions

Overview

While users interact with online services (e.g. search engines, recommender systems, conversational agents), they leave behind fine grained traces of interaction patterns. The ability to understand user behavior, record and interpret user interaction signals, gauge user satisfaction and incorporate user feedback gives online systems a vast treasure trove of insights for improvement and experimentation. More generally, the ability to learn from user interactions promises pathways for solving a number of problems and improving user engagement and satisfaction. Understanding and learning from user interactions involves a number of different aspects — from understanding user intent and tasks, to developing user models and personalization services. A user’s understanding of their need and the overall task develop as they interact with the system. Supporting the various stages of the task involves many aspects of the system, e.g. interface features, presentation of information, retrieving and ranking. Beyond understanding user needs, learning from user interactions involves developing the right metrics and experimentation systems, understanding user interaction processes, their usage context and designing interfaces capable of helping users.

Outline

Phase I: Leveraging User Interactions for Understanding & Extracting User Tasks

• What is a task & why are they important?
• Characterizing Tasks across devices & interfaces: desktop search, digital assistants & voice-only assistants
• Understanding User Tasks in Web Search
  • Extracting Query Intents
  • Search Task Understanding
  • Task extraction
  • Subtask extraction
  • Hierarchies of tasks & subtasks
  • Evaluating task extraction algorithms
• Recommendation Systems
• User intents & goals
• Defining user intents
• Predicting user intents
• Case Studies:
• Intents at Pinterest
• Intents at Spotify

Phase II: Learning User Representations

• Traditional vectors (BoW, Tf-idf)
• Topical profiles + entities of interest
• Collaborative Filtering:
  • Neural CF
  • Coupled CF
  • Deep CF
• Neural representations: embeddings, task specific, multi-view
• Short term vs Long term profiles
• Cold start: cohort based representations
• Multi-task representations
• Multi-view representations

Phase III: Leveraging Interactions for Metrics & Experimentation

• Explicit Feedback
• Crowdsourced judgments
• Lab & field studies
• Data gathering A/B tests
• Implicit Feedback
• Interaction features
• Motifs & sequences
• Satisfaction prediction models
• Sequential models: markov, CRFs
• Semi-supervised models
• Deep sequential models

Format / Slides

• Day 1: Introduction [slides]
• Day 1: Phase I: Leveraging User Interactions for Understanding & Extracting User Tasks [slides]
• Day 2: Phase I (extra): Session Identification Strategies [slides]
• Day 3: Phase II: Learning User Representations [slides]
• Day 4: Phase III: Leveraging Interactions for Metrics & Experimentation [slides]

Related Papers

• KDD 2018: Real-time Personalization using Embeddings for Search Ranking at Airbnb
• IJCAI 2018: CoupledCF: Learning Explicit and Implicit User-item Couplings in Recommendation for Deep Collaborative Filtering
• SIGIR 2018: Understanding and Evaluating User Satisfaction with Music Discovery
• KDD 2018: Perceive Your Users in Depth: Learning Universal User Representations from Multiple E-commerce Tasks
• WSDM 2017: Joint Deep Modeling of Users and Items Using Reviews for Recommendation
• SIGIR 2017: User Interaction Sequences for Search Satisfaction Prediction
• WWW 2017: Predicting Intent Using Activity Logs: How Goal Specificity and Temporal Range Affect User Behavior
• WWW 2017: Neural Collaborative Filtering
• CIKM 2017: Task Embeddings: Learning Query Embeddings using Task Context
• WWW 2017: Identifying User Sessions in Interactions with Intelligent Digital Assistants
• CIKM 2016: Struggling and Success in Web Search
• SIGIR 2016: Predicting User Satisfaction with Intelligent Assistants
• CIKM 2016: Learning to Account for Good Abandonment in Search Success Metrics
• ECIR 2016: Modeling User Interests for Zero-query Ranking
• KDD 2015: Collaborative Deep Learning for Recommender Systems
• WWW 2015: A Multi-View Deep Learning Approach for Cross Domain User Modeling in Recommendation Systems
• SIGIR 2015: Inferring Searcher Attention by Jointly Modeling User Interactions and Content Salience
• SIGIR 2014: Cohort Modeling for Enhanced Personalized Search
• WSDM 2014 Discovering Common Motifs in Cursor Movement Data for Improving Web Search
• SIGIR 2014: Modeling Action-level Satisfaction for Search Task Satisfaction Prediction
• WSDM 2014: Modeling Dwell Time to Predict Click-level Satisfaction
• RecSys 2013: Nonlinear Latent Factorization by Embedding Multiple User Interests
• CHI 2013: Towards estimating web search result relevance from touch interactions on mobile devices
• SIGIR 2013: Mining Touch Interaction Data on Mobile Devices to Predict Web Search Result Relevance
• WWW 2013: Know Your Personalization: Learning Topic level Personalization in Online Services
• CIKM 2013: Building User Profiles from Topic Models for Personalised Search
• SIGIR 2012: Modeling the Impact of Short- and Long-Term Behavior on Search Personalization
• CIKM 2012: Predicting web search success with fine-grained interaction data
• SIGIR 2011: Detecting Success in Mobile Search from Interaction
• AAAI 2010: Collaborative Filtering Meets Mobile Recommendation: A User-Centered Approach
• WWW 2010: Exploring Searcher Interactions for Distinguishing Types of Commercial Intent
• SIGIR 2010: Ready to Buy or Just Browsing? Detecting Web Searcher Goals from Interaction Data
• WSDM 2010: Beyond DCG: User Behavior as a Predictor of a Successful Search Predicting user SAT