*******************************************
Title: "Understanding Upwork's Freelancer and Job search"
Speaker: Alya Abbott and Ioannis Antonellis, Upwork (Elance-oDesk)

Abstract: There is a major challenge in machine learning that has received relatively little attention in conversations on the subject. Before we can engineer features, experiment with models, evaluate and iterate, we need to figure out what we should be modeling in the fist place. What is the best variable to predict given the task at hand and the data available? Where is model development a high priority, and where can we plug in a simple heuristic? How do we combine multiple models into a unified system, and how do we make our models reusable for different tasks? Upwork is the world's largest online workplace, with 10M+ freelancers billing more than $1B annually. With tens of thousands of jobs filled every day in our platform we have the world's best view on what it takes to find the best freelancers for a job as a client or what is your ideal next job as a freelancer. In this talk, we discuss how Upwork has tackled these types of challenges. You will learn why searching for freelancers or jobs is quite different from searching for websites, and hear about our recent efforts to help clients find available, high quality freelancers and freelancers find jobs effectively on our platform. 

*******************************************
Title: "Who is Justin Bieber's Brother, and Other Questions: Crowdsourced Content Curation and Semantic Parsing"
Speaker: Daoud Clarke, Lumi.do

Abstract: I will talk about two unrelated applications of crowdsourcing and try to link them by a tenuous thread. At Lumi, we find the most interesting stories on the web by analysing our users' behaviour, both on the Lumi Android app (think "Tinder for news") and on our web app. Challenges we face include identifying which domains are rich sources of interesting stories, how to get a good mix of trending and personalised content, and how to use social signals within the app. Meanwhile, at the University of Sussex, I've been working on semantic parsing for a crowdsourced database, Freebase, using a crowdsourced dataset of natural language questions and answers. I will talk about how crowdsourcing is changing these areas and what lies ahead.

*******************************************
Title: "Interactive and Deployable Work With Crowds"
Speaker: Jeffrey P. Bigham, Carnegie Mellon University

Abstract: Our notion of crowdsourcing has changed dramatically over the past few years. No longer is crowdsourcing restricted to workflows in which non-experts toil away at decontextualized asynchronous microtasks. Crowdsourcing is tackling complex and interactive work traditionally done by experts. In this talk, I'll highlight some of the complex expert work that we're having the crowd do on our behalf in order to power the next generation of intelligent interactive systems -- from writing an academic paper from a watch, to working with speech recognition to convert speech to text cheaply in real time. I'll discuss how crowdsourcing can amplify what we are able to do on a wide range of problems, and highlight challenges for machine learning in this rich new space.

*******************************************
Title: "Efficient Crowdsourcing at Facebook: The Thousand-Coin Model"
Speaker: Julian Eisenschlos, Facebook's Crowdsourcing Team

Abstract: During this talk we will introduce the product aspects of crowdsourcing at Facebook and dive into some of the algorithmic challenges that we face and how we tackle them. In particular, how we predict the quality of contributions coming from social network users across the globe and how we combine that information to come up with truth estimates. Facebook's Entity Graph powers products such as Checkins, Search, Ads and various recommendation features. In addition to automated methods like imports and web extraction via machine learning, a crowdsourcing system that empowers people to curate the graph is essential to make it representative of its worldwide audience, and to keep it rich, accurate and responsive to changes in the real world. We will start by looking at the experience of contributing on our crowdsourcing platform, looking into the different motivations that people have, and the challenges of keeping them engaged and guiding them to provide quality information in the best possible format. In our setting, like other volunteer based ones, most participants are casual users, making only a few contributions and doing so incidentally in the course of using the service. Previous work often assumes no knowledge of demographic/behavioural information about the contributors and relies on the existence of historical accuracy data, constraining the distribution of questions and answers to estimate the reliability of a user with a single number (the one-coin model). Our case is exactly the opposite, so we take advantage of this scenario to build a classification model assigning weights to the multiple indicators of accuracy for a single contribution. Finally we will show how we accurately aggregate these (possibly uncalibrated) quality scores coming from different sources using random forests and compare the performance against different baselines and standard algorithms on our datasets. 

*******************************************
Title: "How to Achieve Good Quality of Service under Budget Constraints in Crowdsourcing Systems"
Speaker: Long Tran-Thanh, University of Southampton

Abstract: One of the main challenges in crowdsourcing systems is the problem of allocating a set of micro-tasks to the crowd such that the quality of the aggregated outcome reaches a satisfactory level. This is even more difficult when the task allocation process is constrained by a limited monetary budget, and hiring workers (i.e., allocating tasks to them) involves (monetary) payments. In particular, focussing too much on cheap (but low quality) workers will not provide high quality of service (QoS), while hiring users with good QoS might be expensive, and thus, not so many micro-task will be done in total. Given this, we need to find a good trade-off between maximising the aggregated QoS of the outcome and keeping the cost of the whole process under a certain limit.In this talk, I will demonstrate how we managed to achieve this goal in a variety of crowdsourcing systems, including expert crowdsourcing systems (e.g., oDesk), and crowdsourcing platforms with complex workflows (e.g., Soylent).  I will describe a set of efficient task allocation algorithms that can proveably achieve high QoS, and I will also compare their performance with the existing state of the art through extensive simulations and real experiments.

*******************************************
Title: "Time-series Modeling of Annotators in Crowdsourced Data Collection"
Speaker: Matthew Lease, University of Texas, Austin

Abstract: Given the importance of labeled data for training AI systems, crowdsourced data collection has attracted great interest in recent years, including how to optimize trade-offs of scale, quality, and cost. While the performance of individual annotators over time has typically been modeled under an i.i.d. assumption, time-series modeling enables us to track changes in performance over time, enabling better prediction of individual performance and the routing of work to individuals more likely to produce reliable labels. After first describing an effective generative model for temporal modeling of individual performance, I will then describe a more flexible discriminative model which is able to capture additional predictive features. A general challenge in crowdsourced data collection is how to evaluate worker performance (e.g., by comparison to expert gold labels or checking peer-agreement), and cost vs. benefit trade-offs of different alternatives. Recognizing that an individual's performance over time may be non-stationary, we describe two methods for utilizing limited gold labels for training and updating such time-series models. We also explore different ways of performing model updates without gold by using uncertain soft labels. Experiments with a public crowdsourcing dataset demonstrate the feasibility and effectiveness of predicting annotator performance, ultimately both improving the quality of collected data and reducing its cost. This is joint work led by my PhD student, Hyun Joon Jung (linkedin.com/in/hyunvincero).

*******************************************
Title: "Intelligent Control meets Crowdsourcing"
Speaker: Mausam, Indian Institute of Technology Delhi

Abstract: Requesters often experience that crowdsourcing is not suitable for them because of unskilled workers, poor quality control, or other reasons. We believe that crowdsourcing applications can benefit immensely from careful task design, targeted data collection, training of workers, intelligent task routing, and optimized workflows. We argue that decision-theoretic control is an important piece in all these subcomponents. In the talk we discuss several applications of intelligent control for successful crowdsourcing including for quality control, budget-limited active learning with relabeling, worker training, and more.

*******************************************
Title: "Veri.ly: Crowdsourcing Fact-Checking"
Speaker: Victor Naroditskiy, OneMarketData

Abstract: We live in the Information Age or the Disinformation Age. Finding out the truth in the vast amount of contradictory information is becoming increasingly difficult. Can the truth emerge when hundreds of people are asked to contribute evidence to support or disprove and unverified claim? An online platform, veri.ly, enables such collective fact-checking. I will report on the deployments of veri.ly and discuss the use of crowdsourcing for information verification, evaluation of the quality of verification, and for collecting verification requests.

*******************************************