Sweep and point and shoot: phonecam-based interactions for large public displays

Abstract

This paper focuses on enabling interactions with large public displays using the most ubiquitous personal computing device, the mobile phone. Two new interaction techniques are introduced that use the embedded camera on mobile phones as an enabling technology. The "Point & Shoot" technique allows users to select objects using visual codes to set up an absolute coordinate system on the display surface instead of tagging individual objects on the screen. The "Sweep" technique enables users to use the phone like an optical mouse with multiple degrees of freedom and allows interaction without having to point the camera at the display. Prototypes of these interactions have been implemented and evaluated using modern mobile phone technologies. This proof of concept provides a performance baseline and gives valuable insights to guide future research and development. These techniques are intended to inspire and enable new classes of large public display applications.