First International Workshop on Reciprocal Knowledge
Elicitation for Human-Agent Collaboration
in Affiliation with:

UPDATE 21.11.2023: the workshop has been cancelled, but we will give a presentation about reciprocal elicitation at the Inter.HAI workshop !

About the Workshop

Workshop Motivation

Recent research in human-agent interaction focuses on human-agent collaboration where teams of humans and intelligent agents are formed to achieve a shared goal. The young term hybrid intelligence [3] addresses human-agent teams that achieve a superior goal leveraging human creative power and artificial computation power. An important aspect for the success of human-agent teams is co-learning [4,5] which describes the goal to gain experience while performing activities as a team.

Well-established systems that share knowledge with human users are expert systems which make use of extracted expert knowledge to solve real world problems and are a subclass of knowledge-based systems which store explicit domain knowledge ([6], p.18). The conventional approach to build expert systems is described by the knowledge engineering cycle ([6], p.5) where a knowledge engineer applies different knowledge elicitation techniques with domain experts to formalize strategies and domain rules the expert system must adhere to. A comprehensive overview of elicitation techniques was published in 1994 [2]. The addressed technique families observations, interviews, and task analysis are still relevant today and applied in other fields of human-computer interaction like user experience (UX) research (e.g., [1], chapter 11 - task analysis).

The knowledge engineering cycle, however, does not meet the prevailing requirements for modern knowledge-based systems where knowledge must be dynamically updated by users with a different expertise level. In some scenarios, users must provide new knowledge implicitly while working on their active job. Additionally, the user must be empowered to extract knowledge efficiently from the knowledge-based system during active work performance. The mutual need for communication between the user and the knowledge system enforces them to collaborate. We refer to the beneficial application of mutual knowledge elicitation and knowledge provision within a human-agent team as reciprocal knowledge elicitation. The design of reciprocal knowledge elicitation remains unexplored.

Objective and Outcome

This workshop will look to collect and analyze work on knowledge elicitation for human-agent collaboration, bringing together leading researchers from industry and academia in a number of domains. We specifically address researchers from Robotics, Computer Science, Engineering, Cognitive science, and Psychology. We aim to map best practices of knowledge elicitation where originally both knowledge elicitor and domain expert are human, and transfer them to the context of human-agent collaboration.

We aim for the following outcomes

  • A domain-specific overview of attributes to consider for designing (reciprocal) knowledge elicitation processes in the context of human-agent collaboration.
  • An overview of (reciprocal) knowledge elicitation techniques and their eligibility for specific domains in the context of human-agent collaboration.
  • The gathering and networking of researchers and practitioners interested in the field of (reciprocal) knowledge elicitation for human-agent collaboration.

Workshop Structure

The workshop is planned as a full-day event. Due to its participatory design approach, we cannot accept more than eight submissions.

  • The accepted papers will be presented via short 7-minute lightning talks. Additionally, we provide an introduction to the PACT framework ([1], p. 25 - 46).
  • Based on the presented work, teams are formed. The aim of the first group activity is to conduct a PACT analysis for the specific use case and domain.
  • In the second group activity, presented elicitation techniques are evaluated on their fitness regarding the identified attributes of the PACT analysis.
  • In the consolidation phase, identified attributes are collected and a (preliminary) decision matrix for elicitation techniques is created.

The preliminary schedule is shown below:

Time Description
08:45 - 09:00 Welcoming
09:00 - 10:30 Lightning Talks
10:30 - 10:45 Coffee Break
10:45 - 13:00 Group Activity I
13:00 - 14:00 Lunch Break
14:00 - 16:30 Group Activity II
16:30 - 16:45 Coffee Break
16:45 - 18:00 Consolidation


[1] David Benyon. 2019. Designing user experience – A guide to HCI, UX and interaction design (4. ed.). Pearson UK.
[2] Nancy J. Cooke. 1994. Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies 41, 6 (1994), 801–849. ijhc.1994.1083
[3] Dominik Dellermann, Adrian Calma, Nikolaus Lipusch, Thorsten Weber, Sascha Weigel, and Philipp Ebel. 2019. The Future of Human-AI Collaboration: A Taxon- omy of Design Knowledge for Hybrid Intelligence Systems. 24251/HICSS.2019.034
[4] Karel van den Bosch, Tjeerd Schoonderwoerd, Romy Blankendaal, and Mark Neer- incx. 2019. Six Challenges for Human-AI Co-learning. In Adaptive Instructional Systems, Robert A. Sottilare and Jessica Schwarz (Eds.). Springer International Publishing, Cham, 572–589.
[5] Emma M Van Zoelen, Karel Van Den Bosch, and Mark Neerincx. 2021. Becoming team members: Identifying interaction patterns of mutual adaptation for human- robot co-learning. Frontiers in Robotics and AI 8 (2021), 692811.
[6] Donald A. Waterman. 1986. A guide to expert systems. Addison-Wesley, Reading (Mass.)