Papers accepted at CHI 2019

Finger Identification on Capacitive Touchscreens using Deep Learning

Touchscreens enable intuitive mobile interaction. However, touch input is limited to 2D touch locations which makes it challenging to provide shortcuts and secondary actions similar to hardware keyboards and mice. Previous work presented a wide range of approaches to provide secondary actions by identifying which finger touched the display. While these approaches are based on external sensors which are inconvenient, we use capacitive images from mobile touchscreens to investigate the feasibility of finger identification. We collected a dataset of low-resolution fingerprints and trained convolutional neural networks that classify touches from eight combinations of fingers. We focused on combinations that involve the thumb and index finger as these are mainly used for interaction. As a result, we achieved an accuracy of over 92% for a position-invariant differentiation between left and right thumbs. We evaluated the model and two use cases that users find useful and intuitive. We publicly share our data set (CapFingerId) comprising 455,709 capacitive images of touches from each finger on a representative mutual capacitive touchscreen and our models to enable future work using and improving them.

Huy Viet Le, Sven Mayer, and Niels Henze. 2019. Investigating the Feasibility of Finger Identification on Capacitive Touchscreens using Deep Learning. In 24th International Conference on Intelligent User Interfaces (IUI ’19), March 17–20, 2019, Marina del Ray, CA, USA. ACM, New York, NY, USA. https://doi.org/10.1145/3301275.3302295

Papers accepted at CHI 2018

Papers on Estimating Finger Orientation and Reducing Latency using Machine Learning

We will present two papers at the International Conference on Interactive Surfaces and Spaces. For both papers, we trained models that to improve the interaction with smartphones. PredicTouch is a system to reduce touchscreen latency using neural networks and inertial measurement units. With the second paper, we provide a ground truth data set for to estimate finger orientations using capacitive touchscreens recorded with a high-precision motion capture system. Using the data set, we show that a convolutional neural network can outperform approaches proposed in previous work.

Tutorial on Intelligent Mobile User Interfaces @ MobileHCI

Together with Sven and Huy, I’ll give a tutorial on Machine Learning for Intelligent Mobile User Interfaces using TensorFlow. One key feature of TensorFlow includes the possibility to compile the trained model to run efficiently on mobile phones. This enables a wide range of opportunities for researchers and developers. In the tutorial, we teach attendees two basic steps to run neural networks on a mobile phone: Firstly, we will teach how to develop neural network architectures and train them in TensorFlow. Secondly, we show the process to run the trained models on a mobile phone.

CHI 2016 Videos

The Effect of Focus Cues on Separation of Information Layers

Video for our CHI 2016 paper “The Effect of Focus Cues on Separation of Information Layers”, written by Patrick Bader, Niels Henze, Nora Broy and Katrin Wolf.

Impact of Video Summary Viewing on Episodic Memory Recall

Video for our CHI 2016 paper “Impact of Video Summary Viewing on Episodic Memory Recall”, written by Huy Viet Le, Sarah Clinch, Corina Sas, Tilman Dingler, Niels Henze, and Nigel Davies.

CHI 2015 Videos

Modeling Distant Pointing for Compensating Systematic Displacements

Video for our CHI 2015 paper “Modeling Distant Pointing for Compensating Systematic Displacements”, written by Sven Mayer, Katrin Wolf, Stefan Schneegass and Niels Henze.

Subjective and Objective Effects of Tablet’s Pixel Density

Video for our CHI 2015 paper “Subjective and Objective Effects of Tablet’s Pixel Density”, written by Lars Lischke, Sven Mayer, Katrin Wolf, Alireza Sahami Shirazi and Niels Henze.

Text Entry on Tiny QWERTY Soft Keyboards

Video for our CHI 2015 paper “Text Entry on Tiny QWERTY Soft Keyboards” written by Luis A. Leiva, Alireza Sahami, Alejandro Catala, Niels Henze and Albrecht Schmidt from the Universitat Politècnica de València and the University of Stuttgart.

Investigation of Material Properties for Thermal Imaging-Based Interaction

Video for our CHI 2015 paper “Investigation of Material Properties for Thermal Imaging-Based Interaction”, written by Yomna Abdelrahman, Alireza Sahami Shirazi, Niels Henze and Albrecht Schmidt.