IID-4 Interaction and Visualization

IID-4 Interaction and Visualization

Description of the theme

The objective of the theme “interaction and visualization” is to study how can interactive computer systems empower humans, in particular when they interact with data. This multidisciplinary theme combines different areas of computer science (e.g., human-computer interaction, information visualization, machine learning, signal processing) with a range of disciplines such as cognitive psychology, social psychology, cognitive ergonomics, and art. One of its goals is to better understand how to best mobilize computer hardware and software to support human understanding of complex data, with technologies such as virtual and mixed-reality environments, large displays and telepresence rooms, touch- and gesture-based interfaces, tangible interfaces, affective computing systems, and scalable big data infrastuctures. At a more fundamental level, this theme also seeks to better understand how humans perceive, think, learn, communicate, collaborate, solve problems and make decisions with data. Thus the scope of this theme is both fundamental — e.g., modeling the phenomena occurring when humans consume, process or produce data, or using the tools of Information Theory to model, design and evaluate interactive techniques — and applied, with various contexts and application areas such as scientific discovery, product lifetime management, decision support, education and training, and disability. One key focus is on the use of interaction and visualization to promote the transparency and accountability of algorithms as used in machine learning, robotic systems and intelligent systems. Finally, this theme also studies how can data be used to enhance the interaction between humans and computers more broadly, such as when using new machine learning techniques for supporting gesture-based and full-body interaction.

Some hot research topics of the theme

User-centered machine learning

Contributor: Jules Françoise

Summary

Machine Learning (ML) is a powerful tool for building applications that perform complex tasks using computational models estimated from data. While it has become ubiquitous in a wide array of softwares and services, it often remains conceived as a black box that works autonomously on passively collected data. Yet, this viewpoint hides considerable human work of tuning the algorithms, gathering the data, and even deciding what should be modeled in the first place. A growing community at the intersection of Machine Learning and Human Computer Interaction investigates human-centered perspectives on ML that explicitly recognise this human work, reframe machine learning workflows based on situated human working practices, and explore the co-adaptation of humans and systems.

In a traditional machine learning workflow, practitioners collect data, select or engineer features to represent the data, choose a learning algorithm and fine-tune its parameters, and finally assess the quality of the resulting model. This workflow results in long iterations, which limit the user’s ability to interact with the model and affect its results. Interactive machine learning is a research topic at the intersection of ML and HCI, where learning cycles involve more rapid, focused, and incremental model updates than in the traditional machine learning process. Current research directions include:

  • Interactive Machine Teaching: enabling novice users (non machine learning experts) to efficiently train ML and AI algorithms, through new workflows, interaction techniques and visualizations.
  • Collaborative interaction with ML algorithms
  • Understanding and facilitating end-user interaction with ML and AI systems. This is critical in expert domains, for example in the medical field where ML-based systems are developed to assist clinicians, and requires developing visualisations, explanations, or interaction techniques to increase trust and facilitate user understanding of ML predictions.
  • Supporting the workflow of machine learning practitioners through novel visualisations and interactions to help them build more robust models, assess them efficiently in real-life scenarios, and reduce their inherent biases.

Keywords: Interactive Machine Learning, Artificial Intelligence, Human-AI Interaction

Researchers involved or interested

A few references

  • Jules Françoise, Baptiste Caramiaux, and Téo Sanchez. 2021. Marcelle: Composing Interactive Machine Learning Workflows and Interfaces. In The 34th Annual ACM Symposium on User Interface Software and Technology (UIST ’21). Association for Computing Machinery, New York, NY, USA, 39–53. https://doi.org/10.1145/3472749.3474734
  • Françoise, J., & Bevilacqua, F. (2018). Motion-sound mapping through interaction: An approach to user-centered design of auditory feedback using machine learning. ACM Transactions on Interactive Intelligent Systems (TiiS), 8(2), 1-30.
  • Boukhelifa, N., Bezerianos, A., & Lutton, E. (2018). Evaluation of interactive machine learning systems. In Human and Machine Learning (pp. 341-360). Springer, Cham.
  • Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014). Power to the people: The role of humans in interactive machine learning. Ai Magazine, 35(4), 105-120.
  • Dudley, J. J., & Kristensson, P. O. (2018). A review of user interface design for interactive machine learning. ACM Transactions on Interactive Intelligent Systems (TiiS), 8(2), 1-37.

Augmented and mixed reality

Possible contributor: TBD

Summary

(Data exploration in augmented and mixed reality, immersive analytics.)

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

Keywords: Lorem, ipsum, dolor, sit, amet.

Researchers involved or interested

A few references

  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.

✓✓✓ Visualizations beyond the desktop

Contributor: Petra Isenberg

Summary

Visualization fundamentally studies how people work with, react to, understand, or interact with (mostly) digital representations of data. With its focus on people, visualization’s goal is to empower humans to make use of data whenever it is needed the most. Traditionally, visualizations were designed for desktop or laptop computers but both hardware and information needs have changed and new fruistful avenues for research have emerged

  • mobile visualization: Mobile devices and services have a great potential to satisfy the growing need to access data anywhere, at any time. Data is increasingly collected not only in professional but also in personal settings, such as fitness and health tracking, personal finances, gaming, and many others. As such, mobile data visualization can help a broad range of people better access and utilize data.
  • data physicalization: data can not only be represented using digital displays but also using physical and tangible displays. These physicalizations provide tactile, as well as visual metaphors for expressing and experiencing data, and can unlock new analytical insights and emotional responses.
  • large display visualization: large displays offer not only the opportunity to display large quantities of data at once but also the opportunity for multiple spectators or analysts experiencing data together.
  • immersive analytics: immersive analytics is the science of analytical reasoning facilitated by immersive human-computer interfaces. These include in particular virtual reality and augmented reality environments used to display data visualizations.

Keywords: mobile visualization, data physicalization, immersive analytics, non-desktop displays

Researchers involved or interested

A few references

  • Data Physicalization Pierre Dragicevic, Yvonne Jansen, Andrew Vande Moere Jean Vanderdonckt. Springer Handbook of Human Computer Interaction, Springer, 2021, Springer Reference, ISBN 978-3-319-73228-2
  • Visualizing Ranges over Time on Mobile Phones: A Task-Based Crowdsourced Evaluation. Matthew Brehmer, Bongshin Lee, Petra Isenberg, Eun Kyoung Choe. IEEE Transactions on Visualization and Computer Graphics, Institute of Electrical and Electronics Engineers, 2019, 25 (1), pp.619-629.
  • Interaction for Immersive Analytics. Wolfgang Büschel, Jian Chen, Raimund Dachselt, Steven Drucker, Tim Dwyer, Carsten Görg, Tobias Isenberg, Andreas Kerren, Chris North, Wolfgang Stuerzlinger. Immersive Analytics, Springer, pp.95 – 138, 2018.
  • Collaborative Visualization: Definition, Challenges, and Research Agenda Petra Isenberg, Niklas Elmqvist, Jean Scholtz, Daniel Cernea, Kwan-Liu Ma, Hans Hagen. Information Visualization, SAGE Publications, 2011, Special Issue on Information Visualization: State of the Field and New Research Directions, 10 (4), pp.310–326

Perception, cognition, and decision making with visualizations

Possible contributors: TBD

Summary

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Keywords: Lorem, ipsum, dolor, sit, amet.

Researchers involved or interested

A few references

  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.

✓✓✓ Progressive analytics

Contributor: Jean-Daniel Fekete

Summary

We live in an era where data is abundant and growing rapidly; databases storing big data sprawl past memory and computation limits, and across distributed systems. New hardware and software systems have been built to sustain this growth in terms of storage management and predictive computation. However, these infrastructures, while good for data at scale, do not well support exploratory data analysis (EDA) as, for instance, commonly used in Visual Analytics. EDA allows human users to make sense of data with little or no known model on this data and is essential in many application domains, from network security and fraud detection to epidemiology and preventive medicine. Data exploration is done through an iterative loop where analysts interact with data through computations that return results, usually shown with visualizations, which in turn are interacted with by the analyst again. Due to human cognitive constraints, exploration needs highly responsive system response times: at 500ms, users change their querying behavior; past five or ten seconds, users abandon tasks or lose attention. As datasets grow and computations become more complex, response time suffers. To address this problem, a new computation paradigm has emerged in the last decade under several names: online aggregation in the database community; progressive, incremental, or iterative visualization in other communities. It consists of splitting long computations into a series of approximate results improving with time; in this process, partial or approximate results are then rapidly returned to the user and can be interacted with in a fluent and iterative fashion. With the increasing growth in data, such progressive data analysis approaches will become one of the leading paradigms for data exploration systems, but it also will require major changes in the algorithms, data structures, and visualization tools.

Keywords: exploratory data analysis, visualization, visual analytics, database, approximate query processing, progressive data analysis.

Researchers involved or interested

A few references

✓✓✓ Scientific visualization and illustrative visualization

Contributor: Tobias Isenberg

Summary

Illustrative visualization takes inspiration from the long tradition of traditional illustration, in which illustrators have developed numerous excellent techniques to convey complex scientific principles, techniques, data, etc. As such it not only draws from traditional computer graphics but also from non-photorealistic rendering and uses principles like abstaction and emphasis to create visualizations of real datasets. It such has the potential to provide illustration-like visualizations where it would be unfeasible to hire a professional illustrator as well as attempts to create interactively explorable illustration-like visualizations.

Keywords: illustrative visualization, abstraction, emphasis.

Researchers involved or interested

A few references

Alternative input and output modalities for interacting with data

Possible contributor: TBD

Summary

(touch, multiple sensors, haptic,… )

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Keywords: Lorem, ipsum, dolor, sit, amet.

Researchers involved or interested

A few references

  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.

Computer-mediated collaboration

Possible contributor: TBD

Summary

(social touch, colocated and remote)

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Keywords: Lorem, ipsum, dolor, sit, amet.

Researchers involved or interested

A few references

  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.

Human-computer partnership

Possible contributor: TBD

Summary

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Keywords: Lorem, ipsum, dolor, sit, amet.

Researchers involved or interested

A few references

  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.

New interaction techniques

Possible contributor: TBD

Summary

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Keywords: Lorem, ipsum, dolor, sit, amet.

Researchers involved or interested

A few references

  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.
  • Ipsum, Lorem. Lorem ipsum. Proceedings of the 1st International Workshop. 2010.

General references

Andrienko et al. (2020) Big Data Visualization and Analytics: Future Research Challenges and Emerging Applications.

Inria (2020/2021) Livre blanc de l’intelligence artificielle, section 5.9 AI and Human-Computer Interaction (HCI). In press (draft hosted in https://filesender.renater.fr/)

Write your name if you work on / are interested in this theme

Nom Prénom E-mail Affiliation
Dragicevic Pierre pierre.dragicevic@inria.fr Inria
Lecolinet Eric eric.lecolinet@telecom-paristech.fr Télécom Paris
Beaudouin-Lafon Michel mbl@lri.fr LRI, Université Paris-Saclay

Hot topics