Faculty Candidate Seminar

Seeing the Unseen: Data-Driven 3D Scene Understanding for Robot Vision

Shuran SongPh.D. CandidatePrinceton University

Intelligent robots require advanced vision capabilities to perceive and interact with the real physical world. While computer vision has made great strides in recent years, its predominant paradigm still focuses on analyzing image pixels to infer 2D output representations (bounding boxes, segmentation, etc.), which remain far from sufficient for real-world robotics applications.

In this talk, I will advocate the use of complete 3D scene representations that enable intelligent systems to not only recognize what is seen (e.g. Am I looking at a chair?) but also predict contextual information about the complete 3D environment beyond visible surfaces (e.g. What could be behind the table? Where should I look to find an exit?). As examples, I will present a line of my recent works that demonstrate the power of these representations through amodal 3D object detection (Sliding Shape and Deep Sliding Shapes), analyzing and synthesizing 3D scenes (Semantic Scene Completion), and predicting semantic and 3D structure outside the image field of view (Im2Pano3D). Finally, I will discuss some ongoing efforts on how these 3D scene representations can further enable and benefit from real-world robotic interactions (in the Amazon Robotic Challenge), shifting the way we view computer vision problems from the perspective of a passive observer to that of an active explorer.

Shuran Song is a fifth-year Ph.D. candidate in Computer Science at Princeton University, advised by Thomas Funkhouser. Before that, she received a BEng. in Computer Science from HKUST in 2013 with the highest honor. During her Ph.D., she spent time working at Microsoft Research and Google Daydream. Her research interests lie at the intersection of computer vision, computer graphics, and robotics. She was awarded the Facebook Fellowship in 2014, Siebel Scholar in 2016, Wallace Fellowship in 2017, and Princeton SEAS Award for Excellence in 2017. She was named Princeton Innovation's 25 Under 25, and was part of the MIT-Princeton team for the Amazon Robotics Challenge, winning the 3rd place in 2016 and 1st place (stow task) in 2017.

Sponsored by


Faculty Host

Clayton Scott and Chad Jenkins