• Location: ICML 2015 Workshop, Lille, France
• Room: Faidherbe
• Posters: Boards are located in the "Espace Flandre"
• Date: July 10, 2015
• Schedule outline is available here
• Title+Abstract for invited talks are available here
• Resources: See more information on CrowdML site
• Sponsors: We acknowledge the support of Microsoft Research and University of Southampton - ORCHID project for partially sponsoring CrowdML at ICML'15.
• Contact: For any questions, please email icml15crowd AT gmail

How can we build systems that combine the intelligence of humans and the computing power of machines for solving challenging scientific and engineering problems?

The goal is to improve the performance of complex human-powered systems by making them more efficient, robust, and scalable.

Machine Learning with strategic agents. Machine learning algorithms (for instance, active learning by querying experts, information gathering from sensors) are typically designed to interact with non-strategic components (sensor nodes, machines, or non-strategic people). The human-powered systems present a big paradigm shift as these components are being replaced by strategic agents, for example, workers in crowdsourcing systems aiming to maximize their profits, students or participants as learning entities, smartphones of the people as sensor nodes, and so on. This poses a number of challenges and interesting research questions, such as using statistical techniques to understand, model and learn human behavior, and designing robust ML systems that can deal with intrinsic noise and strategic behavior of these components controlled by human agents.

Incentives, pricing mechanisms and budget allocation. How can we design the right incentive structure and pricing policies for participants that maximize the satisfaction of participants as well as utility of the job requester for a given budget? How can techniques from machine learning, economics and game theory be used to learn optimal pricing policies and to infer optimal incentive designs?

Task decomposition and knowledge aggregation. How can complex crowdsourcing tasks be decomposed into simpler micro-tasks that can be performed by individuals or small groups with relatively little effort? How can we design models and algorithms to effectively aggregate responses and knowledge, especially for complex tasks?

Learning by participants and peer evaluation. How can we use insights from machine learning to build tools for training and teaching the participants (workers/students) in the crowdsourcing systems for carrying out difficult tasks and in MOOCs? How can this training be actively adapted based on the skills or expertise of the participants and by tracking the learning process? Peer evaluation schemes including peer prediction and information elicitation, and their applications to peer-grading in MOOCs, peer review, crowdsourced data labeling, incentivizing effort, etc. are also relevant topics.

Social aspects and collaboration. With ever-increasing time on the Internet being spent on online social networks, there is a huge opportunity to elicit useful contributions from users at scale, by carefully designing tasks. How can online social networks be used to create tasks with a gamification component and engage users in useful activities? How can systems exploit the underlying social ties of the participants to create incentives for users to collaborate?

Human-in-the-loop ML systems. How can we build practical systems that seamlessly integrate machine and human intelligence, a.k.a. human-in-the-loop ML systems? Machine learning algorithms can help the crowdsourcing component to manage workflows and control workers' qualities, while the crowds can be used to handle tasks that are difficult for machines to adaptively boost the performance of machine learning algorithms. Active learning and decision-theoretic techniques can be helpful to actively adapt the workflow of tasks that are given to workers.

Open theoretical questions, challenges, and novel applications. The scale of the Internet, the fast-evolving landscape of technology, and the increase of mobile computing is creating tremendous opportunities and challenges for the design of new human-powered systems. What are the open research questions, emerging trends and novel applications at the intersection of crowdsourcing and machine learning? This workshop will encourage visionary position papers to discuss this further.