NIPS ’13 Workshop on Crowdsourcing: Theory, Algorithms and Applications

Abstracts of Invited Talks

Michael Bernstein: Crowd-Powered Systems. [slides]

Abstract: Crowd-powered systems combine computation with human intelligence, drawn from large groups of people connecting and coordinating online. These hybrid systems enable applications and experiences that neither crowds nor computation could support alone.

Unfortunately, crowd work is error-prone and slow, making it difficult to incorporate crowds as first-order building blocks in software systems. I introduce computational techniques that decompose complex tasks into simpler, verifiable steps to improve quality, and optimize work to return results in seconds. These techniques advance crowdsourcing into a platform that is reliable and responsive to the point where crowds can be used in interactive systems.

In this talk, I will present two crowd-powered systems to illustrate these ideas. The first, Soylent, is a word processor that uses paid micro-contributions to aid writing tasks such as text shortening and proofreading. Using Soylent is like having access to an entire editorial staff as you write. The second system, Adrenaline, is a camera that uses crowds to help amateur photographers capture the exact right moment for a photo. It finds the best smile and catches subjects in mid-air jumps, all in realtime. These systems point to a future where social and crowd intelligence are central elements of interaction, software, and computation.

Bio: Michael Bernstein is an Assistant Professor of Computer Science at Stanford University, where he directs the Human-Computer Interaction group and is a Robert N. Noyce Family Faculty Scholar. His research in human-computer interaction focuses on the design of crowdsourcing and social computing systems. This work has received Best Paper awards and nominations at premier venues in human-computer interaction and social computing (ACM UIST, ACM CHI, ACM CSCW, AAAI ISWSM). Michael was awarded the George M. Sprowls Award for best doctoral thesis in Computer Science at MIT. He holds Ph.D. and masters degrees in Computer Science from MIT, and a B.S. in Symbolic Systems from Stanford University.

Evgeniy Gabrilovich: Crowdsourcing knowledge, one billion facts at a time

ABSTRACT: According to the Knowledge Principle formulated by Lenat and Feigenbaum (1991), computer programs need to know a great deal about the world in order to perform well in complex tasks. However, building world-scale knowledge bases cannot yet be fully automated, and is still critically dependent on humans for knowledge collection and curation. In this talk, we will describe our work on using crowdsourcing to make Google's Knowledge Graph (and its extension, Knowledge Vault) more comprehensive and accurate. In particular, we will discuss Quizz, a new gamified crowdsourcing system that simultaneously assesses the knowledge of users and acquires new knowledge from them. Quizz actively tries to identify knowledgeable users on the Internet by running advertising campaigns, effectively leveraging ‘‘for free’’ the targeting capabilities of existing, publicly available, ad placement services. We demonstrate that the cost of our approach is below that of hiring workers through paid-crowdsourcing platforms, while offering the additional advantage of giving access to billions of potential users all over the planet, and being able to reach users with specialized expertise that is not typically available through existing labor marketplaces.

Bio: Dr. Evgeniy Gabrilovich is a senior staff research scientist at Google, where he works on knowledge discovery from the web. Prior to joining Google in 2012, he was a director of research and head of the natural language processing and information retrieval group at Yahoo! Research. Evgeniy is an ACM Distinguished Scientist (2012), and is a recipient of the 2010 Karen Sparck Jones Award for his contributions to natural language processing and information retrieval. He served as an area chair or senior program committee member at numerous major conferences, including SIGIR, WWW, WSDM, AAAI, IJCAI, ACL, EMNLP, CIKM, ICDM and ICWSM. He has organized a number of workshops and taught multiple tutorials at SIGIR, ACL, WWW, WSDM, ICML, IJCAI, AAAI, CIKM and EC. Evgeniy earned his PhD in computer science from the Technion - Israel Institute of Technology.

Devavrat Shah: Collaborative Decision Making

The process of making decisions amongst stake holders with potentially different views and limited information is omnipresent – examples include democratic policy making, business operations and even decision of dining venue amongst friends. Efficient automation of collaborative decision making requires developing (a) useful human-interface to seek information, (b) statistical model to capture human uncertainty, and (c) efficient inference algorithm. We have develop such a framework in the context of two scenarios: micro-task crowd sourcing and rank aggregation. We shall discuss the associated statistical models, algorithms and their information optimality. This is based on joint works with Ammar Ammar, David Karger, Sahand Negahban and Sewoong Oh.

Arpita Ghosh: Incentive design for crowdsourcing: A game-theoretic approach. [slides]

The Web is increasingly centered around the collective effort of the crowds, with contributions ranging from user-generated content and social media to Citizen Science projects and crowd-based peer-grading and peer-learning in online education. But every crowdsourcing based system relies on users actually participating and making high-quality contributions to function effectively. How can we design systems so that self-interested users — with their own costs and benefits to participation — are properly incentivized to participate and contribute with high effort?

We discuss a game-theoretic framework for incentive design in crowdsourcing, where potential contributors are modeled as self-interested agents who strategically choose whether or not to participate, and how much effort to expend, in response to the incentives offered by the system. We illustrate the game-theoretic approach via the problem of learning the qualities of online content from viewer feedback: here, a learning algorithm needs to not only quickly identify the best contributions, but also simultaneously create incentives for attention-motivated users to make high-quality contributions, leading to a multi-armed bandit problem where the number and success probabilities of the arms of the bandit are endogenously determined in response to the learning algorithm.