3rd ICML 2021 Workshop on Human in the Loop Learning
Recent years have witnessed the rising need for machine learning systems that have humans in the learning loop. Such systems can be applied to computer vision, natural language processing, robotics, and human-computer interaction. Creating and running such systems call for interdisciplinary research of artificial intelligence, machine learning, and cognitive science, which we abstract as Human in the Loop Learning (HILL). The HILL workshop aims to bring together researchers and practitioners working on the broad areas of HILL, ranging from the interactive/active learning algorithms for real-world decision-making systems (e.g., autonomous driving vehicles, robotic systems, etc.), lifelong learning systems that retain knowledge from different tasks and selectively transfer knowledge to learn new tasks over a lifetime, models with strong explainability, as well as human-inspired learning. The HILL workshop continues the previous effort to provide a platform for researchers from interdisciplinary areas to share their recent research. In this year’s workshop, a special feature is to encourage the exploration of human-inspired learning.
July 24
Virtual Conference
Call for Papers
​
We welcome high-quality submissions on algorithms and system designs in the broad area of human in the loop learning. A few (non-exhaustive) topics of interest include:
​
-
The topics of HILL include but are not limited to:
-
Interactive/Active machine learning algorithms for autonomous decision-making systems,
-
Lifelong learning systems that learn a sequence of tasks and leverage their shared structure to enable knowledge transfer over a lifetime,
-
Online learning and active learning,
-
Comparison of human in the loop learning and label-efficient learning,
-
Psychology driven human concept learning,
-
Explainable AI,
-
Human-inspired learning,
-
Design, testing, and assessment of interactive systems for data analytics,
-
Model understanding tools (debugging, visualization, introspection, etc.).
These topics span a variety of scientific disciplines and application domains like machine learning, human-computer interaction, cognitive science, and robotics. It is an opportunity for scientists in these disciplines to share their perspectives, discuss solutions to common problems and highlight the challenges in the field to help guide future research. The target audience for the workshop includes people who are interested in using machines to solve problems by having a human be an integral part of the learning process.
​
We invite submissions of full papers, as well as works-in-progress, position papers, and papers describing open problems and challenges. While original contributions are preferred, we also invite submissions of high-quality work that has recently been published in other venues or is concurrently submitted. We encourage creative ML approaches, as well as interdisciplinarity and perspectives from outside traditional ML. Papers should be 4-8 pages in length (excluding references) formatted using the ICML template. All the submissions should be anonymous. The accepted papers are allowed to get submitted to other conference venues.
​
Papers can be submitted through CMT:
​
hhttps://cmt3.research.microsoft.com/HILL2021
​
​
Important Dates
​
Submission deadline: 27th June 2021 (23:59 AoE)
​
Acceptance notification: 7th, July 2021 (23:59 AoE)
​
Workshop Date: 24th July
Accepted Papers on 2nd HILL Workshop at ICML 2021
Paper
PreferenceNet: Encoding Human Preferences in Auction Design
Neehar Peri (University of Maryland)*; Michael J Curry (University of Maryland College Park); Samuel Dooley (University of Maryland); John P Dickerson (University of Maryland)
Paper
IADA: Iterative Adversarial Data Augmentation Using Formal Verification and Expert Guidance
Ruixuan Liu (Carnegie Mellon University)*; Changliu Liu (Carnegie Mellon University)
Paper
Machine Teaching with Generative Models for Human Learning
Michael Doron (The Broad Institute)*; Hussein Mozannar (Massachusetts Institute of Technology); David Sontag (MIT); Juan Caicedo (Broad Institute)
Paper
Supp
Differentiable Learning Under Triage
​
Nastaran Okati (MPI-SWS)*; Abir De (IIT Bombay); Manuel Gomez Rodriguez (MPI-SWS)
High Frequency EEG Artifact Detection with Uncertainty via Early Exit Paradigm
Paper
Lorena Qendro (University of Cambridge)*; Alex Campbell (University of Cambridge); Pietro Lió (University of Cambridge); Cecilia Mascolo (University of Cambridge)
Paper
Improving Human Decision-Making with Machine Learning
Hamsa Bastani (Wharton); Osbert Bastani (University of Pennsylvania); Wichinpong Sinchaisri (Berkeley Haas)*
Paper
Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos
Haoyu Xiong (Shanghai Qizhi Institute)*; Quanzhou Li (University of Toronto); Yun-Chun Chen (University of Toronto ); Homanga Bharadhwaj (University of Toronto, Vector Institute); Samarth Sinha (University of Toronto, Vector Institute); Animesh Garg (University of Toronto, Vector Institute, Nvidia)
Paper
To Trust or Not to Trust a Regressor: Estimating and Explaining Trustworthiness of Regression Predictions
Kim de Bie (University of Amsterdam)*; Ana Lucic (University of Amsterdam); Hinda Haned (University of Amsterdam)
Paper
Interpretable Machine Learning: Moving From Mythos to Diagnostics
alerie Chen (Carnegie Mellon University)*; Jeffrey Li (University of Washington); Joon Sik Kim (Carnegie Mellon University); Gregory Plumb (); Ameet Talwalkar (CMU)
Paper
Shared Interest: Large-Scale Visual Analysis of Model Behavior by Measuring Human-AI Alignment
Angie W Boggust (MIT CSAIL)*; Benjamin Hoover (IBM Research); Arvind Satyanarayan (MIT CSAIL); Hendrik Strobelt (IBM Research)
Paper
CF-GNNExplainer: Counterfactual Explanations for Graph Neural Networks
Ana Lucic (University of Amsterdam)*; Maartje A ter Hoeve (University of Amsterdam); Gabriele Tolomei (University of Rome); Maarten de Rijke (University of Amsterdam & Ahold Delhaize); Fabrizio Silvestri (Sapienza, University of Rome)
Paper
Supp
Personalizing Pretrained Models
Mina Khan (Massachusetts Institute of Technology (MIT) Media Lab)*; Advait P Rane (BITS Pilani-K. K. Birla Goa Campus); Srivatsa P (National University of Singapore); Asadali Hazariwala (BITS Pilani Goa); Shriram Chenniappa (BITS Pilani Goa); Pattie Maes (Massachusetts Institute of Technology (MIT) )
Convergence of a Human-in-the-Loop Policy-Gradient Algorithm With Eligibility Trace Under Reward, Policy, and Advantage Feedback
Paper
Ishaan K Shah (Brown University); David M Halpern (Brown University)*; Michael L. Littman (Brown University); Kavosh Asadi (Brown University)
Paper
Effect of Combination of HBM and Certainty Sampling onWorkload of Semi-Automated Grey Literature Screening
JINGHUI LU (University College Dublin)*; Maeve Henchion (Teagasc Agriculture and Food Development Authority); Brian Mac Namee (University College Dublin )
Paper
A Simple Baseline for Batch Active Learning with Stochastic Acquisition Functions
Andreas Kirsch (University of Oxford)*; Sebastian Farquhar (University of Oxford); Yarin Gal (University of Oxford)
Paper
Active Learning under Pool Set Distribution Shift and Noisy Data
Andreas Kirsch (University of Oxford)*; Tom Rainforth (University of Oxford); Yarin Gal (University of Oxford)
Paper
Explaining Reinforcement Learning Policies through Counterfactual Trajectories
Julius Frost (Boston University)*; Olivia Watkins (UC Berkeley); Eric M Weiner (Harvey Mudd College); Pieter Abbeel (UC Berkeley); Trevor Darrell (UC Berkeley); Bryan Plummer (Boston University); Kate Saenko (Boston University)
Paper
Differentially Private Active Learning with Latent Space Optimization
Sen-ching S Cheung (University of Kentucky)*; Xiaoqing Zhu (Cisco); Herb Wildfeuer (Cisco); Chongruo Wu; Wai-tian Tan (Cisco)
Explicable Policy Search via Preference-Based Learning under Human Biases
Ze Gong (Arizona State University)*; Yu Zhang (ASU)
Paper
Paper
Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap
Gokul Swamy (Carnegie Mellon University)*; Sanjiban Choudhury (Aurora Innovation); Drew Bagnell; Steven Wu (Carnegie Mellon University)
Paper
On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models.
Zeyad Emam (University of Maryland, College Park); Andrew Kondrich (Scale AI); Sasha Harrison (Sasha Harrison); Felix Lau (Scale AI); Yushi Wang (Scale AI); Aerin Kim (Scale AI)*; Elliot Branson (Scale AI)
Paper
ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
Yuanfei Wang (Peking University)*; Fangwei Zhong (Peking University); Jing Xu (Peking University); Yizhou Wang (PKU)
Accelerating the Convergence of Human-in-the-Loop Reinforcement Learning with Counterfactual Explanations
Jakob Karalus (Ulm University )*; Felix Lindner (Ulm University)
Paper
Less is more: An Empirical Analysis of Model Compression for Dialogue
Ahmed O BARUWA (KPMG)*
Paper
Mitigating Sampling Bias and Improving Robustness in Active Learning
Ranganath Krishnan (Intel Labs)*; Alok Kumar Sinha (Intel); Nilesh A Ahuja (Intel); Mahesh Subedar (Intel); Omesh Tickoo (Intel); Ravi Iyer (Intel)
Paper
Paper
GCExplainer: Human-in-the-Loop Concept-based Explanations for Graph Neural Networks
Lucie Charlotte Magister (University of Cambridge)*; Dmitry Kazhdan (University of Cambridge); Vikash Singh (Alphabet X); Pietro Lió (University of Cambridge)
Paper
Interpretable Video Transformers in Imitation Learning of Human Driving
Andrew Dai (Trinity College Dublin, Department of Civil, Structural and Environmental Engineering)*; Wenliang Qiu (Trinity College Dublin, Department of Civil, Structural and Environmental Engineering); Bidisha Ghosh (Trinity College Dublin, Department of Civil, Structural and Environmental Engineering)
Speakers of 2nd HILL Workshop at ICML 2020
Professor at Princeton University
Professor at University of Toronto
Assistant professor at Stanford University
Professor at University of Toronto
Professor at Tsinghua University
Assistant professor at UC Berkeley
Postdoc Researcher at Facebook AI Research (FAIR)
Assistant professor at UC Berkeley
Research Scientist at Carnegie Mellon University
Associate professor at Carnegie Mellon University
Professor at
University of Tübingen