2nd ICML 2020 Workshop on Human in the Loop Learning
Organizers: Shanghang Zhang , Xin Wang, Fisher Yu, Jiajun Wu, Trevor Darrell
Recent years have witnessed the rising need for learning agents that can interact with humans. Such agents usually involve applications in computer vision, natural language processing, human computer interaction, and robotics. Creating and running such agents call for interdisciplinary research of artificial intelligence, machine learning, and software engineering design, which we abstract as Human in the Loop Learning (HILL). HILL is a modern machine learning paradigm of significant practical and theoretical interest. For HILL, models and humans engage in a two-way dialog to facilitate more accurate and interpretable learning. It includes active learning, where the learners choose which examples to label and achieve better accuracy with less data compared to the classical approach of passively observing labeled data; It also includes the explainable learning, where the human doesn't merely tell the machine whether its predictions are correct, but provides reasons in a form that is meaningful to both parties.
The workshop aims to bring together researchers and practitioners working on the broad areas of human in the loop learning, ranging from the interactive/active learning algorithm designs for real-world decision making systems (e.g., autonomous driving vehicles, robotic systems, etc.), models with strong explainability, as well as interactive system designs (e.g., data visualization, annotation systems, etc.). It provides an opportunity for scientists to share their perspectives, discuss solutions to these questions, and highlight the challenges in the field to help guide future research. In particular, we aim to elicit new connections among these diverse fields, identifying theory, tools and design principles tailored to practical machine learning workflows. In this year’s HILL workshop, we emphasize on the interactive/active learning algorithms for real-world decision making systems as well as learning algorithms with strong explainability. We continue the previous effort to provide a platform where researchers can discuss approaches that bridge the gap between humans and machines and get the best of both worlds.
July 18
Virtual Conference
Speakers
FEATURED TALKS
Professor at Princeton University
Professor at University of Toronto
Assistant professor at Stanford University
Professor at University of Toronto
Professor at Tsinghua University
Assistant professor at UC Berkeley
Postdoc Researcher at Facebook AI Research (FAIR)
Assistant professor at UC Berkeley
Research Scientist at Carnegie Mellon University
Associate professor at Carnegie Mellon University
Professor at
University of Tübingen
Agenda
July 18
Accepted Papers
Zoom meeting links for the poster sessions：
https://docs.google.com/spreadsheets/d/1gegBSjv8Sf766Mzh01yczfjf4ExUl4-4X8wUa4mlGxQ/edit?usp=sharing
or
(Same content with the google doc)
If there are Zoom meetings you cannot access, please comment on the google doc beside these meetings' links
Paper
Poster
A Linear Bandit for Seasonal Environments
Giuseppe Di Benedetto (Oxford Univerisity); Vito Bellini (Amazon); Giovanni Zappella (Amazon)
Online Ride-Sharing Pricing with Fairness
Yupeng Li (University of Toronto/Shenzhen Research Institute of Big Data)*; Mengjia Xia (Cornell University); Dacheng Wen (The University of Hong Kong); Cheng Zhang (Didi Chuxing); Meng Ai (Didi Chuxing); Qun (Tracy) Li (DiDi)
Deep Active Learning: Unified and Principled Method for Query and Training
Changjian Shui (Université Laval)*; Fan Zhou (Laval University); Christian Gagné (Université Laval); Boyu Wang (University of Western Ontario)
GLAD: Localized Anomaly Detection via Human-in-the-Loop Learning
Md Rakibul Islam (Washington State university)*; Shubhomoy Das (School of EECS, Washington State University, Pullman); Janardhan Rao Doppa (Washington State University); Sriraam Natarajan (UT Dallas)
Human-Centric Efficiency Improvements in Image Annotation for Autonomous Driving
Frédéric Ratle (Samasource)*; Martine Bertrand (Samasource)
Online Learning for Distributed and Personal Recommendations - a Fair approach
Martin Tegnér (IKEA Retail, Oxford-Man Institute, University of Oxford)*
Yet Another Study on Active Learning and Human Pose Estimation
Sinan Kaplan (Lappeenranta University of Technology)*; Lasse Lensu (Lappeenranta University of Technology)
Program Synthesis with Pragmatic Communication
Yewen Pu (MIT)*; Marta Kryven (Massachusetts Institute of Technology); Kevin M Ellis (MIT); Joshua Tenenbaum (MIT); Armando Solar-Lezama (MIT)
Preference learning along multiple criteria: A game-theoretic perspective
Kush Bhatia (UC Berkeley)*; Ashwin Pananjady (UC Berkeley); Peter Bartlett (); Anca Dragan (EECS Department, University of California, Berkeley); Martin Wainwright (UC Berkeley)
Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations
Sarath Sreedharan (Arizona State University)*; Utkarsh Soni (Arizona State University); Mudit Verma (Arizona State University); Siddharth Srivastava (Arizona State University); Subbarao Kambhampati (Arizona State University)
Interactive Segmentation of RGB-D Indoor Scenes using Deep Learning
Maximilian Ruethlein (Friedrich-Alexander University Erlangen-Nuernberg)*; Franz Koeferl (Friedrich-Alexander University Erlangen-Nuernberg); Wolfgang Mehringer (Friedrich-Alexander University Erlangen-Nuernberg); Bjoern Eskofier (Friedrich-Alexander University Erlangen-Nuernberg)
Interactive learning of cognitive programs
Sunayana Rane (MIT)*; Miguel Lázaro-Gredilla (Vicarious AI); Dileep George (Vicarious )
The Need for Standardised Explainability
Othman Benchekroun (Dathena)*; Adel Rahimi (Dathena); Qini Zhang (Dathena); Tetiana Kodliuk (Dathena)
Quick Question: Interrupting Users for Microtasks with Reinforcement Learning
Bo-Jhang Ho (UCLA); Bharathan Balaji (Amazon)*; Mehmet Koseoglu (UCLA); Sandeep Singh Sandha (University of California - Los Angeles); Siyou Pei (UCLA); Mani Srivastava (UC Los Angeles)
Adwait Sahasrabhojanee (USRA/NASA Ames); David Iverson (NASA Ames); shawn r wolfe (); Kevin Bradner (NASA Ames); Nikunj Oza (NASA Ames)*
Active Learning Strategies to Reduce Anomaly Detection False Alarm Rates
SCRAM: Simple Checks for Realtime Analysis of Model Training for Non-Expert ML Programmers
Eldon Schoop (University of California, Berkeley)*; Forrest Huang (University of California, Berkeley); Bjoern Hartmann (University of California, Berkeley)
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos
Laura M Smith (UC Berkeley)*; Nikita Dhawan (UC Berkeley); Marvin Zhang (UC Berkeley); Pieter Abbeel (UC Berkeley); Sergey Levine (UC Berkeley)
Personalized Stress Detection with Self-supervised Learned Features
Stefan Matthes (Fortiss GmbH); Zhiwei Han (fortiss GmbH)*; Tianming Qiu (fortiss GmbH); Bruno Michel (IBM Zurich Research Lab); Sören Klinger (fortiss GmbH); Hao Shen (fortiss GmbH); Yuanting Liu (fortiss GmbH); Bashar Altakrouri (IBM Deutschland GmbH)
Metric-Free Individual Fairness in Online Learning
Yahav Bechavod (Hebrew University of Jerusalem)*; Steven Wu (University of Minnesota); Christopher Jung (University of Pennsylvania)
Bias in Multimodal AI: Testbed for Fair Automatic
Alejandro Peña (Universidad Autonoma de Madrid); Ignacio Serna (Universidad Autonoma de Madrid); Aythami Morales (Universidad Autonoma de Madrid)*; Julian Fierrez (Universidad Autonoma de Madrid)
Explanation Augmented Feedback in Human-in-the-Loop Reinforcement Learning
Lin Guan (Arizona State University)*; Mudit Verma (Arizona State University); Subbarao Kambhampati (Arizona State University)
Not all Failure Modes are Created Equal: Training Deep Neural Networks for Explicable (Mis)Classification
Alberto Olmo (Arizona State University)*; Sailik Sengupta (Arizona State University); Subbarao Kambhampati (Arizona State University)
Battlesnake Challenge: A Multi-agent Reinforcement Learning Playground with Human-in-the-loop
Jonathan Chung (Amazon Web Services)*; Runfei Luo (Amazon Web Services); Xavier Raffin (Amazon Web Services); Scott Perry (Amazon Web Services)
Faster Human-Machine Collaboration Bounding Box Annotation Framework Based on Active Learning
Minzhe Liu (Nanjing University)*; LI DU (Nanjing University); Yuan Du (Nanjing University); Ruofan Guo (Nanjing University); Xiaoliang Chen (University of California, Irvine)
Combining Human and Machine Intelligence to Assess Stroke Rehabilitation Exercises
Min Hun Lee (Carnegie Mellon University)*; Daniel Siewiorek (Carnegie Mellon University); Asim Smailagic (Carnegie Mellon University); Alexandre Bernardino (Instituto Superior Técnico); Sergi Bermudez (University of Madeira)
Personalized Size Recommendations with Human in the Loop
Leonidas Lefakis (Zalando)*; Evgenii Koriagin (Zalando SE); Julia Lasserre (Zalando Research); Reza Shirvany (Zalando SE)
A Prospective Human-in-the-Loop Experiment using Reinforcement Learning with Planning for Optimizing Energy Demand Response
Lucas Spangher (U.C. Berkeley)*; Manan Khattar (University of California at Berkeley); Akash Gokul (University of California at Berkeley); Akaash Tawade (University of California at Berkeley); Adam Bouyamourn (University of California at Berkeley); Alex R Devonport (U.C. Berkeley); Costas J. Spanos (University of California at Berkeley)
Learning Interpretable Models for Black-Box Agents
Pulkit Verma (Arizona State University)*; Siddharth Srivastava (Arizona State University)
Assisted Robust Reward Design
Jerry Zhi-Yang He (EECS Department, University of California, Berkeley)*; Anca Dragan (EECS Department, University of California, Berkeley)
Better Transferability with Attribute Attention for Generalized Zero-Shot Learning
Ruofan Guo (Nanjing University)*; LI DU (Nanjing University); Yuan Du (Nanjing University); Minzhe Liu (Nanjing University); Xiaoliang Chen (University of California, Irvine)
Human Explanation-based Learning for Machine Comprehension
Qinyuan Ye (University of Southern California)*; Xiao Huang (University of Southern California); Elizabeth Boschee (University of Southern California); Xiang Ren (University of Southern California)
Soliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems
Hao-Fei Cheng (University of Minnesota)*; Paige Bullock (Kenyon College); Alexandra Chouldechova (CMU); Steven Wu (University of Minnesota); Haiyi Zhu (Carnegie Mellon University)
Improve black-box sequential anomaly detector relevancy with limited user feedback
Chris Kong (Amazon research); Lifan Chen (Amazon research); Ming Chen (Amazon research); Laurent Callot (Amazon research)*; Parminder Bhatia (Amazon)
Feature Expansive Reward Learning: Rethinking Human Input
Andreea Bobu (UC Berkeley)*; Marius Wiggert (UC Berkeley); Claire Tomlin (UC Berkeley); Anca Dragan (EECS Department, University of California, Berkeley)
Call for Papers
We welcome high-quality submissions on algorithms and system designs in the broad area of human in the loop learning. A few (non-exhaustive) topics of interest include:
-
Active/Interactive machine learning algorithms for autonomous decision-making systems,
-
Online learning and active learning,
-
Psychology driven human concept learning,
-
Explainable AI,
-
Systems for online and interactive learning algorithms,
-
Systems for collecting, preparing, and managing machine learning data,
-
Design, testing and assessment of interactive systems for data analytics,
-
Model understanding tools (debugging, visualization, introspection, etc).
These topics span a variety of scientific disciplines and application domains like machine learning, human-computer interaction, cognitive science, and robotics. It is an opportunity for scientists in these disciplines to share their perspectives, discuss solutions to common problems and highlight the challenges in the field to help guide future research. The target audience for the workshop includes people who are interested in using machines to solve problems by having a human be an integral part of the learning process.
We invite submissions of full papers, as well as works-in-progress, position papers, and papers describing open problems and challenges. While original contributions are preferred, we also invite submissions of high-quality work that has recently been published in other venues or is concurrently submitted. We encourage creative ML approaches, as well as interdisciplinarity and perspectives from outside traditional ML. Papers should be 4-8 pages in length (excluding references) formatted using the ICML template. All the submissions should be anonymous. The accepted papers are allowed to get submitted to other conference venues.
Papers can be submitted through CMT:
https://cmt3.research.microsoft.com/HILL2020
Important Dates
Submission deadline: 25th June 2020 (23:59 AoE)
Acceptance notification: 10th, July 2020 (23:59 AoE)
Workshop Date: 18th July
