University of Michigan researchers create screen protection system to fend off shoulder surfers

Most of us use our phones in public without worry, but accessing private information when out and about can be a risky undertaking. Increasingly, thieves are resorting to a relatively simple means of stealing data from unknowing strangers: shoulder surfing.

From public transit to waiting rooms to elevators and beyond, shoulder surfers exploit physical proximity in crowded places to view device screens and obtain potentially sensitive data, such as passwords, PINs, financial information, and more, simply by peering over your shoulder. In one recent high-profile case, a man in the UK had £22,000 stolen after a shoulder surfing incident.

In response to this threat, researchers at the University of Michigan have developed Eye-Shield, an innovative screen protection system that obscures images and text on your phone and other devices when viewed from a distance. Eye-Shield was designed by Brian Tang, doctoral candidate in computer science and engineering, and Kevin and Nancy O’Connor Professor of Computer Science Kang G. Shin.

“The key question we are trying to answer in this project,” said Tang, “is how we can improve on existing technologies to protect users’ information while browsing the web, watching videos, or doing virtually any activity on their phones, tablets, or laptops.”

Previous solutions have been ineffective, inconvenient, or limited in scope. Some involve the application of a physical privacy film to your device, which can’t be turned off or easily removed, offers only limited protection, and in many cases prevents the use of a screen protector. Others take the form of apps that target specific functions, including one that obscures numbers by overlaying low- and high-frequency images and another that replaces text with hard-to-read handwriting.

Eye-Shield, on the other hand, is designed to live in your device as a free, built-in option that you can toggle on and off depending on your needs. The program leverages our visual perception of contrast to blur text and images at a distance.

“Let’s say you have two light sources,” said Tang. “Up close, it is easy to see both light sources. But if you look at them from a distance, they merge into a single light source.”

Using this phenomenon, the researchers developed a checkered grid made up of contrasting colored boxes. The use of this pattern causes pixels to merge when viewed from a certain distance or angle, making the device screen appear blurry to potential shoulder surfers but not to the person using the device.

Eye-Shield is composed of a complex set of algorithms that determine the right size of these checkered grid squares based on various parameters, such as distance from the screen, text font size, pixel density, and more. These calculations ensure appropriate protection without substantially altering user experience.

Tests with a group of over 20 participants showed that Eye-Shield was highly effective in protecting device screens from shoulder surfing. The researchers placed participants at varying distances and angles from a device to replicate a shoulder surfing situation and then asked them to identify the content, consisting of both text and images.

Through these tests, they found that Eye-Shield reduced shoulder surfers’ ability to recognize text and images by roughly 60%. On an unprotected screen, participants standing at a distance of 41 inches were able to recognize over 80% of screen content. With Eye-Shield, this number decreased to less than 25%. This number is even lower, just over 15%, for text alone. 

In addition to fending off shoulder surfers, Eye-Shield is able to preserve the screen’s appearance for the intended user, an issue that existing solutions have largely failed to overcome. Eye-Shield also functions in real time with little to no effect on speed or performance.

“With Eye-Shield deployed, we found that participants were able to recognize 90% of the content on a screen when viewed up close without having seen it before,” said Tang. “Using your own phone, when you have some context for what you’re seeing, these numbers will only be higher.”

Tang and Prof. Shin have filed a patent for Eye-Shield, and their ultimate goal is to work with device designers and manufacturers to integrate their screen protection program into commonly used operating systems. In the medium term, Tang and Prof. Shin aim to make further improvements to Eye-Shield’s system by experimenting with contrast and other parameters to boost its efficacy even further. 

“We hope that this work will lay a foundation for further advances to improve the safety of devices and protect people’s private information,” said Tang.

This work is reported in Tang and Shin’s paper titled Eye-Shield: Real-Time Protection of Mobile Device Screen Information from Shoulder Surfing, appearing at the 2023 USENIX Security Symposium.