Touchscreens enable intuitive mobile interaction. However, touch input is limited to 2D touch locations which makes it challenging to provide shortcuts and secondary actions similar to hardware keyboards and mice. Previous work presented a wide range of approaches to provide secondary actions by identifying which finger touched the display. While these approaches are based on external sensors which are inconvenient, we use capacitive images from mobile touchscreens to investigate the feasibility of finger identification. We collected a dataset of low-resolution fingerprints and trained convolutional neural networks that classify touches from eight combinations of fingers. We focused on combinations that involve the thumb and index finger as these are mainly used for interaction. As a result, we achieved an accuracy of over 92% for a position-invariant differentiation between left and right thumbs. We evaluated the model and two use cases that users find useful and intuitive. We publicly share our data set (CapFingerId) comprising 455,709 capacitive images of touches from each finger on a representative mutual capacitive touchscreen and our models to enable future work using and improving them.
We will present two papers at the International Conference on Interactive Surfaces and Spaces. For both papers, we trained models that to improve the interaction with smartphones. PredicTouch is a system to reduce touchscreen latency using neural networks and inertial measurement units. With the second paper, we provide a ground truth data set for to estimate finger orientations using capacitive touchscreens recorded with a high-precision motion capture system. Using the data set, we show that a convolutional neural network can outperform approaches proposed in previous work.
There will be one billion smartphone users in 2013 and most of them will need some sort of text entry. To help people to enter text on mobile devices we aimed at studying how people type with a large number of participants. Therefore, we developed a typing game that records how users touch on the standard Android keyboard to investigate users’ typing behaviour. We published the typing game Type It! on the Android Market. The game got installed by 72,945 players and enabled us to collect 47,770,625 keystrokes from around the world.
Using the data we identified three approaches to improve text entry on mobile phones. As we found a systematic skew in users’ touch distribution we derived a function that compensates this skew by shifting touch events. In addition, we changed the keys’ labels by shifting them upwards and visualize the position where users touch the keyboard. By updating the game we conducted an experiment that investigates the effect of the three approaches. Results based on 6,603,659 further keystrokes and 13,013 installations show that visualizing the touched positions using a simple dot decreases the error rate of the Android keyboard by 18.3% but also decreases the speed by 5.2% with no positive effect on learnability. The Android keyboard outperforms the control condition but the constructed shift function further improves the performance by 2.2% and decreases the error rate by 9.1%. We argue that the shift function can improve existing keyboards at no costs.
Recently I analysed the data we collect using Hit It! one of our Android games that has been installed more than 350,000 times. To play the game users have to tap on circles that randomly appear on the screen. While the game is played we record the user’s behaviour and send it back to our server. In particular, we record the positions that are tapped by the player’s finger. Looking at the hit positions relative to the presented circles we did some pretty nifty things. However, one would expect that the positions that the players tap are somewhat evenly distributed across the screen when combining a serious number of taps.
Interestingly, for the GT-i9003 the distribution looks strange. For the images below we took a random sample consisting of data from 40,000 devices from our data set and extracted the data produced by the GT-i9003 and by the GT-i9000 (a regular Samsung Galaxy S). Data for the GT-i9003 has been produced by 157 devices resulting in 170,205 taps. Data for the GT-i9000 has been produced by 2,321 devices resulting in 3,689,138 taps.
The pixels’ colours show how often a particular pixel has been tapped. Green pixels are tapped often while red pixels are tapped less often. Black pixels are never tapped at all. Due to the nature of the game players have to tap on the screen’s centre more often. For the GT-i9003 we see that half the pixels are NEVER tapped at all. Considering the amount of data and the small difference between the two devices (see below) this obviously can’t be by chance.
The touch-part of the GT-i9003’s touchscreen seems to have only half the resolution of the regular Galaxy S. Furthermore, the hardware or software deals with this limitation in a strange way. The pattern is interesting. There are two rows of pixels that can be tapped followed by two rows that are never tapped. I assume that if the finger taps an “untappable” row the input is mapped to one of the adjacent rows. Considering the somewhat strange pattern and that I can get sub-pixel resolution from the Android API I assume that it is a software problem rather than a hardware issue or probably a combination of hard- and software.
The GT-i9003 is the only device that returns that pattern. If someone owns a GT-i9003 I would be very interested to hear if the effect results in practical issues. But there are also other devices that might have some issues. E.g. there are some Optimus Ones that deserve further investigation (too much variance in the distribution) and I would also like to look at the Kyocera Zio.
Type It! is a game for the Android platform that is all about speed and quick fingers. It challenges (and hopefully improves) your texting abilities. You have to touch and type as fast as you can to see if you can beat all levels. The player’s task is to enter the words that appear as fast as possible. The faster they are the more points they get. Players might improve their dexterity by trying to be the fastest guy in the high score.
This game is part of our research about the touch performance on mobile devices and also part of my work as a PhD student. While users play the game we measure where they hit the screen and how fast they are. By combining this information with the position of the keyboard we can estimate how easy each key is to touch. Based on this data we are hopefully able to predict user’s performance with different keys and character sequences. We plan to derive an according model and this model could possibly be used to improve the virtual keyboards of current smartphones.
We hope that we can collect data from thousands of players. That would enable us to derive information that is valid not only for a small number of people but for every user. We are, however, not interested in you contact list, browsing history, or phone number. Okay – if you are good looking I might be interested in your phone number but I don’t want to collect such data automatically ;). In general we don’t want or need data that enables identifying individuals. Thus, we do not collect those things or other personal information.
Type It! is available for Android 2.1 and above. You can have a look at users’ comments and the game’s description on AppBrain or install it directly on your Android phone from the Market.