The first application of GPT-4 is a “virtual volunteer” for the visually impaired

Image sources: Be my Eye

OpenAI has introduced its latest powerful AI model to the world, GPT-4, and refreshingly, the first thing they claimed in the new system was to help visually impaired people. Be My Eyes, which allows blind and partially sighted people to ask sighted people to describe what their phone sees, is getting a “virtual volunteer” to offer AI-powered help at any time.

We have written about Be My Eyes many times since its launch in 2015, and of course the rise of computer vision and other devices that help the visually impaired to navigate everyday life has played a prominent role in its history. But the app itself can only do so much, and the core function has always been to get a volunteer helping hand who could look through the phone’s camera and provide detailed descriptions or instructions.

The new version of the application integrates the multimodel capabilities of GPT-4, that is, its ability not only to chat intelligibly, but also to examine and understand the received images.

Users can send images through the app to an AI-powered virtual volunteer who will answer any question about the image and provide instant visual assistance for a variety of tasks.

For example, if a user sends a picture of the inside of their refrigerator, the Virtual Volunteer can not only correctly identify what’s inside, but also extrapolate and analyze what can be made with those ingredients. The tool can then offer you a number of recipes for these ingredients and send you step-by-step instructions on how to prepare them.

But the video accompanying the description is more illustrative. In it, Be My Eyes user Lucy shows off the app that helps her with a bunch of things live. If you’re not familiar with the screen reader’s rapid-fire cotton, you might miss some of the dialogue, but you can describe what a garment looks like, identify a plant, read a map, translate a label, control an empty treadmill at the gym, and tell it which buttons to press. at the machine. (You can watch the video below.)

Be My Eyes is a virtual volunteer

It shows very succinctly how unfriendly much of our urban and commercial infrastructure is to people with visual impairments. And it also shows how useful GPT-4’s multimodal chat can be under the right circumstances.

There’s no doubt that human volunteers will continue to be useful to users of the Be My Eyes app – they can’t be replaced, they only raise the bar when needed (and indeed can be called in immediately if the AI’s response isn’t good enough).

The Be My Eyes team works closely with OpenAI and its community to define and manage its capabilities as development continues.

Currently, the feature is in closed beta for a “small portion” of Be My Eyes users, which will expand in the coming weeks. “We hope to make Virtual Volunteer widely available in the coming months,” the team writes. “Like our existing volunteering service, this tool is free for all blind and partially sighted communities using the Be My Eyes app.”

Considering how quickly ChatGPT was brought on board to provide services for enterprise SaaS platforms and other rather prosaic applications, it’s nice to see the novelty immediately getting to work and helping people. You can read more about GPT-4 here.