ICML ’13 Workshop: Machine Learning Meets Crowdsourcing

Abstracts of Invited Talks

Jeffrey P. Bigham : Crowd Agents: Interactive Crowd-Powered Systems in the Real World

Over the past few years, we have been developing and deploying interactive crowd-powered systems that help people get things done in their everyday lives. For instance, VizWiz answers visual questions for blind people in less than a minute, Legion drives robots in response to natural language commands, Chorus supports consistent dialog between end users and the crowd, and Scribe converts streaming speech to text in less than five seconds. Overall, thousands of people have engaged with these systems, providing an interesting look at how end users interact with crowd work in their everyday lives. These systems have collectively informed a new model for real-time crowd work that I call “crowd agents,” which is proving to be especially useful for building interactive crowd-powered systems. In this model, a diverse and changing crowd – the kind easily recruited on the web – is made to act as a single high-quality actor through interface support and computational mediation of each individual’s work. These systems allow us to deploy truly intelligent interactive systems today, and present challenging problems for machine learning going forward to support and eventually replace the humans in the loop.

Yiling Chen: Financial Incentives and Crowd Work

Online labor markets such as Amazon Mechanical Turk (MTurk) have emerged as platforms that facilitate the allocation of productive effort across global economies. Many of these markets compensate workers with monetary payments. We study the effects of performance-contingent financial rewards on work quality and worker effort in MTurk via two experiments. We find that the magnitude of performance-contingent financial rewards alone affects neither quality nor effort. However, when workers working on two tasks of the same type in a sequence, the change in the magnitude of the reward over the two tasks affects both. In particular, both work quality and worker effort increase (alternatively decrease) as the reward increases (alternatively decreases) for the second task. This suggests the existence of the anchoring effect on workers’ perception of incentives in MTurk and that this effect can be leveraged in workflow design to increase the effectiveness of financial incentives.

Panagiotis G. Ipeirotis : Rewarding Crowdsourced Workers

We describe techniques for rewarding workers in a crowdsourcing setting. We describe a real-time monetary payment scheme that rewards workers according to their quality, in the presence of uncertainty in quality estimation, while at the same time guaranteeing stable (or increasing) salaries. We report experimental results indicating that the proposed scheme encourages long-term engagement, avoiding churn, and avoiding the common problem of adverse selection and moral hazard. We also describe a set of non-monetary, psychological schemes that actively discourage low-quality workers from participating in tasks. We finish showing that mice and crowdsourced workers are not that different after all.

Edith Law : Mixed-Expertise Crowdsourcing

To date, most of the research in human computation focuses on tasks that can be performed by any person with basic perceptual capabilities and common sense knowledge. In this talk, I will discuss new directions towards mixed-expertise crowdsourcing, where the crowd consists of people with drastically different motivations, levels and domains of expertise, as well as availabilities. I will illustrate the new opportunities and challenges in mixed-expertise crowdsourcing, by outlining existing work and describing my two ongoing projects – Curio, a micro-task marketplace for crowdsourcing scientific tasks, and SimplyPut, a crowdsourcing platform for improving health literacy through the collaborative summarization of medical information.

Mark Steyvers: Aggregating Human Judgments in Combinatorial Problems

We analyze the collective performance of individuals in combinatorial problems involving the rankings of events and items (e.g. “what is the order of US presidents?”) as well as traveling salesperson and minimum spanning tree problems. We compare situations in which a group of individuals independently answer these questions with an iterated learning environment in which individuals pass their solution to the next person in a chain. We introduce Bayesian information aggregation models for both the independent and information-sharing environments and treat the collective group knowledge as a latent variable that can be estimated from the observed judgments across individuals. The models allow for individual differences in expertise and confidence in other individuals’ judgments. Initial results suggest that information-sharing environments lead to better collective performance despite the fact that information-sharing increases correlations between judgments. In addition, the models’ estimates of expertise are more indicative of actual performance than the users’ self-rated expertise. Finally, we study situations where the same individual solves the same problem at different points in time. We show that the consistency in answers across repeated problems provides an additional signal to estimate expertise.