Deep-learning technique reveals “invisible” objects in the dark
Small imperfections in a wine glass or tiny creases in a contact lens can be tricky to make out, even in good light. In almost total darkness, images of such transparent features or objects are nearly impossible to decipher. But now, engineers at MIT have developed a technique that can reveal these “invisible” objects, in the dark.
In a study published today in Physical Review Letters, the researchers reconstructed transparent objects from images of those objects, taken in almost pitch-black conditions. They did this using a “deep neural network,” a machine-learning technique that involves training a computer to associate certain inputs with specific outputs — in this case, dark, grainy images of transparent objects and the objects themselves.
The team trained a computer to recognize more than 10,000 transparent glass-like etchings, based on extremely grainy images of those patterns. The images were taken in very low lighting conditions, with about one photon per pixel — far less light than a camera would register in a dark, sealed room. They then showed the computer a new grainy image, not included in the training data, and found that it learned to reconstruct the transparent object that the darkness had obscured.
The results demonstrate that deep neural networks may be used to illuminate transparent features such as biological tissues and cells, in images taken with very little light.
“In the lab, if you blast biological cells with light, you burn them, and there is nothing left to image,” says George Barbastathis, professor of mechanical engineering at MIT. “When it comes to X-ray imaging, if you expose a patient to X-rays, you increase the danger they may get cancer. What we’re doing here is, you can get the same image quality, but with a lower exposure to the patient. And in biology, you can reduce the damage to biological specimens when you want to sample them.”
Barbastathis’ co-authors on the paper are lead author Alexandre Goy, Kwabena Arthur, and Shuai Li.
Deep dark learning
Neural networks are computational schemes that are designed to loosely emulate the way the brain’s neurons work together to process complex data inputs. A neural network works by performing successive “layers” of mathematical manipulations. Each computational layer calculates the probability for a given output, based on an initial input. For instance, given an image of a dog, a neural network may identify features reminiscent first of an animal, then more specifically a dog, and ultimately, a beagle. A “deep” neural network encompasses many, much more detailed layers of computation between input and output.
A researcher can “train” such a network to perform computations faster and more accurately, by feeding it hundreds or thousands of images, not just of dogs, but other animals, objects, and people, along with the correct label for each image. Given enough data to learn from, the neural network should be able to correctly classify completely new images.
Deep neural networks have been widely applied in the field of computer vision and image recognition, and recently, Barbastathis and others developed neural networks to reconstruct transparent objects in images taken with plenty of light. Now his team is the first to use deep neural networks in experiments to reveal invisible objects in images taken in the dark.
“Invisible objects can be revealed in different ways, but it usually requires you to use ample light,” Barbastathis says. “What we’re doing now is visualizing the invisible objects, in the dark. So it’s like two difficulties combined. And yet we can still do the same amount of revelation.”
The law of light
The team consulted a database of 10,000 integrated circuits (IC), each of which is etched with a different intricate pattern of horizontal and vertical bars.
“When we look with the naked eye, we don’t see much — they each look like a transparent piece of glass,” Goy says. “But there are actually very fine and shallow structures that still have an effect on light.”
Instead of etching each of the 10,000 patterns onto as many glass slides, the researchers used a “phase spatial light modulator,” an instrument that displays the pattern on a single glass slide in a way that recreates the same optical effect that an actual etched slide would have.
The researchers set up an experiment in which they pointed a camera at a small aluminum frame containing the light modulator. They then used the device to reproduce each of the 10,000 IC patterns from the database. The researchers covered the entire experiment so it was shielded from light, and then used the light modulator to rapidly rotate through each pattern, similarly to a slide carousel. They took images of each transparent pattern, in near total darkness, producing “salt-and-pepper” images that resembled little more than static on a television screen.
The team developed a deep neural network to identify transparent patterns from dark images, then fed the network each of the 10,000 grainy photographs taken by the camera, along with their corresponding patterns, or what the researchers called “ground-truths.”
“You tell the computer, ‘If I put this in, you get this out,’” Goy says. “You do this 10,000 times, and after the training, you hope that if you give it a new input, it can tell you what it sees.”
“It’s a little worse than a baby,” Barbastathis quips. “Usually babies learn a bit faster.”
The researchers set their camera to take images slightly out of focus. As counterintuitive as it seems, this actually works to bring a transparent object into focus. Or, more precisely, defocusing provides some evidence, in the form of ripples in the detected light, that a transparent object may be present. Such ripples are a visual flag that a neural network can detect as a first sign that an object is somewhere in an image’s graininess.
But defocusing also creates blur, which can muddy a neural network’s computations. To deal with this, the researchers incorporated into the neural network a law in physics that describes the behavior of light, and how it creates a blurring effect when a camera is defocused.
“What we know is the physical law of light propagation between the sample and the camera,” Barbastathis says. “It’s better to include this knowledge in the model, so the neural network doesn’t waste time learning something that we already know.”
Sharper image
After training the neural network on 10,000 images of different IC patterns, the team created a completely new pattern, not included in the original training set. When they took an image of the pattern, again in darkness, and fed this image into the neural network, they compared the patterns that the neural network reconstructed, both with and without the physical law embedded in the network.
They found that both methods reconstructed the original transparent pattern reasonably well, but the “physics-informed reconstruction” produced a sharper, more accurate image. What’s more, this reconstructed pattern, from an image taken in near total darkness, was more defined than a physics-informed reconstruction of the same pattern, imaged in light that was more than 1,000 times brighter.
The team repeated their experiments with a totally new dataset, consisting of more than 10,000 images of more general and varied objects, including people, places, and animals. After training, the researchers fed the neural network a completely new image, taken in the dark, of a transparent etching of a scene with gondolas docked at a pier. Again, they found that the physics-informed reconstruction produced a more accurate image of the original, compared to reproductions without the physical law embedded.
“We have shown that deep learning can reveal invisible objects in the dark,” Goy says. “This result is of practical importance for medical imaging to lower the exposure of the patient to harmful radiation, and for astronomical imaging.”
This research was supported, in part, by the Intelligence Advanced Research Projects Activity and Singapore’s National Research Foundation.