Facebook now lets you search for photos by describing their contents

Facebook AI now lets you search a photo by describing the contents of the image or using keywords.

This is different than searching the photos in which you are tagged or pictures that are labeled i.e text-word searching. Many times users forget to label a photo or add a description and it becomes difficult for them to search later. With Facebook’s new AI search, users just have to describe the contents of the photo e.g, “black shirt”, “water park” or “walking to..” etc., and Facebook will identify the pictures through visual search.

“We’ve pushed computer vision to the next stage with the goal of understanding images at the pixel level. This helps our systems do things like recognize what’s in an image, what type of scene it is, if it’s a well-known landmark, and so on. This, in turn, helps us better describe photos for the visually impaired and provide better search results for posts with images and videos.

More than 200 visual models have been trained and deployed on Lumos by dozens of teams, for purposes such as objectionable-content detection, spam fighting, and automatic image captioning. The applications are wide-reaching, with everyone from our Connectivity Labs to Search to the Accessibility team using the technology.”

Lumos is Facebook’s platform that is built for image and video understanding. When photos are uploaded, this platform not only stores them on Facebook but also memorize their contents with deep machine learning and image understanding technology e.g, where the photo was taken, what are people doing in that, what colors etc. This is mainly helpful for visually impaired people.

“Until recently, these captions described only the objects in the photo. Today we’re announcing that we’ve added a set of 12 actions, so image descriptions will now include things like “people walking,” “people dancing,” “people riding horses,” “people playing instruments,” and more.”

Facebook says that it is just the beginning of the visual search and it may also bring this feature for video contents in future.

https://www.facebook.com/Engineering/videos/10154900873827200/