A recent study conducted by a team of researchers from the Massachusetts Institute of Technology (MIT) and Rice University used artificial intelligence (AI) to examine how reflections in a myriad of objects can be turned into “cameras”. This study holds the potential to improve seeing objects that are around corners and out of view and could be especially beneficial to autonomous vehicles (i.e., self-driving cars).
Researchers from MIT and Rice University have used artificial intelligence to develop a computer vision method called ORCa that converts reflections in glossy objects into “cameras”, thus allowing someone to observe the world much like they were peering through the “lenses” of everyday objects such as a ceramic coffee mug or a metallic paper weight. (Credit: Kushagra Tiwary, Ramesh Raskar, et. al)
“We have shown that any surface can be converted into a sensor with this formulation that converts objects into virtual pixels and virtual sensors. This can be applied in many different areas,” said Kushagra Tiwary, who is a PhD student in the MIT Media Lab and lead author of the study.
Contrary to what’s seen on television and in movies where reflections can be enhanced from almost any surface to reveal explicit details, such as on crime shows, the reality is much different, and trying to “enhance” reflections actually make them much more distorted than they were before. This distortion depends on a variety of factors, including the shape, colors, and textures of the object that houses the reflection. Along with that, reflections only cast a two-dimensional image that also distorts the depth perception, as well.
“In real life, exploiting these reflections is not as easy as just pushing an enhance button. Getting useful information out of these reflections is pretty hard because reflections give us a distorted view of the world,” said Akshat Dave, who is a PhD student in the Computational Imaging Lab at Rice University, and a co-author on the study.
For the study, the researchers developed ORCa: Glossy Objects as Radiance-Field Cameras, which captures images from multiple angles then uses artificial intelligence to create a 3D model from the reflective object’s vantage point. In the end, ORCa is capable of modeling scenes in what is known as a 5D radiance field, meaning it can use data such as light direction from every point in the scene being developed.
“It was especially challenging to go from a 2D image to a 5D environment. You have to make sure that mapping works and is physically accurate, so it is based on how light travels in space and how light interacts with the environment. We spent a lot of time thinking about how we can model a surface,” said Tiwary.
Upon comparing ORCa with other reflection-modeling methods, the researchers discovered that ORCa surpassed other methods based on separating real-world colors from the reflected colors, as well as capturing a greater number of textures and geometry of the reflecting object.
“Consistently, with ORCa, it not only estimates the environment accurately as a 5D image, but to achieve that, in the intermediate steps, it also does a good job estimating the shape of the object and separating the reflections from the object texture,” said Dave.
Going forward, the researchers aspire to apply ORCa to drone imaging to digitally rebuild scenes from the ground, along with experimenting with ORCa and shadows to improve its modeling capabilities and combining reflection from more than one object, as well.
“Estimating specular reflections is really important for seeing around corners, and this is the next natural step to see around corners using faint reflections in the scene,” said Ramesh Raskar, who is an associate professor of media arts and sciences and leader of the MIT Camera Culture Group.
Could this novel method someday be used to improve technology in self-driving cars and other applications? Only time will tell, and this is why we science!
Sources: arXiv, MIT News, GitHub
As always, keep doing science & keep looking up!