Harvard smartphone app guides the blind faster and with fewer collisions

If you have standard human vision (even with the help of glasses), you’re probably adept at finding things, people, and paths to them. But if you have poor vision, or none at all, countless tasks that are simple for people with full vision become difficult, dangerous, or even impossible. Fortunately, many new high-tech tools offer assistance, including robotic guide dogs, a cane-bot (that looks a bit like a little R2D2 with giant wheels), multiple types of smart glasses with descriptive audio, and even a haptic vest for blind dogs.

Adding to these technological advances is Mobilio, an artificial intelligence application for smartphones developed by Harvard University that uses personalized sensors and beeps that offers real-time navigation to desired locations while directing users away from obstacles and dangers.

in your Nature Biomedical Engineering In the article, PhD student Raymond Liu and assistant professor of bioengineering Patrick Slade explain how 14 test subjects who used the Mobilio app with a cane were able to get to their destinations 13% faster (and with 41% fewer indoor collisions) than those who used a cane with Google Maps (Google Maps is not detailed enough for safe use by the visually impaired).

The many advantages of Mobilio arise first from the sociological needs assessment and experience of more than 100 people with limited or no vision, and whose responses demonstrated three core needs: continuous guidance along straight paths; precise, turn-by-turn, real-time navigation; and reliable obstacle detection.

The app’s semantic segmentation datasets use images of pedestrians, not drivers, and are therefore more accurate and relevant to people walking on sidewalks and crosswalks rather than driving on roads. As Liu explains, “more data is needed that comes from the perspective of humans walking on sidewalks or walking outdoors.”

Because his own older brother is blind, Liu is acutely aware of the need to consult with those his technology seeks to help. “Having real people test your device is absolutely crucial, because it’s impossible to predict how someone will react to your device,” Liu says. “Especially in the case of blindness and visual impairment, which is a huge and diverse set of disabilities.”

Another advantage of Mobilio is that by taking advantage of widespread smartphone adoption, the app saves users from having to purchase expensive or hard-to-find equipment. Instead, it employs the smartphone’s camera, GPS, inertial detection, and (when present) LiDAR. To this mix, Liu and Slade added an innovative computer vision model that analyzes live camera video to provide real-time safe pathing along a developing route by prompting users with customizable audio cues in tune with the user’s actual walking pattern.

“Our system uses a smartphone’s sensors, GPS information, and the person’s actual movement to create what is essentially a small autonomous vehicle plan for how to guide it from where it is to where it wants to go,” Slade says. Liu notes that a key advance for Mobilio was its performance in a standard smartphone, rather than relying on expensive and inaccessible special devices with faster processors and greater memory.

The team at Harvard’s John A. Paulson School of Engineering and Applied Sciences (SEAS) plans to test Mobilio in more real-world environments to evaluate how well it can adapt to users’ daily routines, habits, and preferences.

Source: Harvard University