By Wayne Gillam / UW ECE News
UW ECE and Physics Professor Arka Majumdar and his students have collaborated with Princeton University to build a new type of compact camera engineered for computer vision. Their prototype (shown above) uses optics for computing, significantly reducing power consumption and enabling the camera to identify objects at the speed of light. Photo by Ilya Chugunov, courtesy of Princeton University
Collaboration can be a beautiful thing, especially when people work together to create something new. Take, for example, a longstanding collaboration between Arka Majumdar, a UW professor in electrical and computer engineering and physics, and Felix Heide, an assistant professor of computer science at Princeton University. Together, they and their students have produced some eye-popping and groundbreaking research, including shrinking a camera down to the size of a grain of salt while still capturing crisp, clear images.
Now, the pair is building on this work, recently publishing a paper in Science Advances that describes a new kind of compact camera engineered for computer vision — a type of artificial intelligence that allows computers to recognize objects in images and video. Majumdar and Heide’s research prototype uses optics for computing, significantly reducing power consumption and enabling the camera to identify objects at the speed of light. Their device also represents a new approach to the field of computer vision.
“This is a completely new way of thinking about optics, which is very different from traditional optics. It’s end-to-end design, where the optics are designed in conjunction with the computational block,” Majumdar said. “Here, we replaced the camera lens with engineered optics, which allows us to put a lot of the computation into the optics.”
Majumdar (left) and Felix Heide, an assistant professor of computer science at Princeton University. Photos by Ryan Hoover (UW ECE) and Steven Schultz (Princeton University)
“There are really broad applications for this research, from self-driving cars, self-driving trucks and other robotics to medical devices and smartphones. Nowadays, every iPhone has AI or vision technology in it,” added Heide, who was the principal investigator and senior author of the Science Advances paper. “This work is still at a very early stage, but all of these applications could someday benefit from what we are developing.”
Heide and his students at Princeton provided the design for the camera prototype, which is a compact, optical computing chip. Majumdar contributed his expertise in optics to help engineer the camera, and he and his students fabricated the chip in the Washington Nanofabrication Laboratory. The UW side of this multi-institutional research team included Johannes Froech, a UW ECE postdoctoral scholar, and James Whitehead (Ph.D. ‘22), who was a UW ECE doctoral student in Majumdar’s lab when this research took place.
Replacing a camera lens with engineered optics
Instead of using a traditional camera lens made out of glass or plastic, the optics in this camera relies on layers of 50 meta-lenses — flat, lightweight optical components that use microscopic nanostructures to manipulate light. These meta-lenses fit into a compact, optical computing chip (shown above), which was fabricated in the Washington Nanofabrication Laboratory by Majumdar and his students. Photo by Ilya Chugunov, courtesy of Princeton University
Instead of using a traditional camera lens made out of glass or plastic, the optics in this camera relies on layers of 50 meta-lenses — flat, lightweight optical components that use microscopic nanostructures to manipulate light. The meta-lenses also function as an optical neural network, which is a computer system that is a form of artificial intelligence modeled on the human brain. This unique approach has a couple of key advantages. First, it’s fast. Because much of the computation takes place at the speed of light, the system can identify and classify images more than 200 times faster than neural networks that use conventional computer hardware, and with comparable accuracy. Second, because the optics in the camera rely on incoming light to operate, rather than electricity, the power consumption is greatly reduced.
“Our idea was to use some of the work that Arka pioneered on metasurfaces to bring some of those computations that are traditionally done electronically into the optics at the speed of light,” Heide said. “By doing so, we produced a new computer vision system that performs a lot of the computation optically.”
Majumdar and Heide say that they intend to continue their collaboration. Next steps for this research include further iterations, evolving the prototype so it is more relevant for autonomous navigation in self-driving vehicles. This is an application area they both have identified as promising. They also plan to work with more complex data sets and problems that take greater computing power to solve, such as object detection (locating specific objects within an image), which is an important feature for computer vision.
“Right now, this optical computing system is a research prototype, and it works for one particular application,” Majumdar said. “However, we see it eventually becoming broadly applicable to many technologies. That, of course, remains to be seen, but here, we demonstrated the first step. And it is a big step forward compared to all other existing optical implementations of neural networks.”
The article, “Spatially varying nanophotonic neural networks,” was published Nov. 8 in Science Advances. In addition to Majumdar, Froech, Whitehead, and Heide, co-authors include Kaixuan Wei, Xiao Li, Praneeth Chakravarthula, and Ethan Tseng. The work was supported by the National Science Foundation, the Defense Advanced Research Projects Agency, a Packard Foundation Fellowship, a Sloan Research Fellowship, a Disney Research Award, a Bosch Research Award, an Amazon Science Research Award, a Google Ph.D. Fellowship, and Project X, an innovation fund of Princeton’s School of Engineering and Applied Science.