MIT researchers build a neural network to reproduce clear images from blurred images and videos

Scientists at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have been working on a machine learning model capable of recovering details from blurry video frames to recreate clear images. The system, called the visual deprojection model, uses a convoluted neural network (CNN) to scan thousands of pairs of images, called projections, one of the sources blurry picture and the other a low-quality picture of the same scene, to pick out fine details and join them together. The result is a new image that shows the scene as if it wasn’t blurry. “It’s almost like magic that we’re able to recover this detail,” said CSAIL postdoc Guha Balakrishnan, lead author of the paper.

This is obviously an oversimplification of the process and in reality, it is far more complex. In effect, this technology is similar to creating the image in reverse by learning the patterns of pixels produced and what usually causes them. A ‘variational autocoder’ then analyzes the resultant image and checks to see how well the CNN matched the new image to what was actually in the blurred image. In this way, the system learns which approach works and what doesn’t, automatically making itself better in the process. The more images the CNN processes, the better it gets at deciphering blurry images correctly, with minimal input from the researchers themselves. 

The applications of this new technology are extremely promising. The researchers think the best place to use their model is the medical field to create 3D scans similar to CT scans. These new scans could be a lot cheaper than expensive alternatives like MRI’s, which require costly machinery and technicians. The new technology could create very high information images from things like X-rays, which don’t cost much compared to MRI’s and CT scans. “If we can convert X-rays to CT scans, that would be somewhat game-changing,” said Balakrishnan. “You could just take an X-ray and push it through our algorithm and see all the lost information.”