Qixing Huang

Hybrid 3D Representations

Machine learning algorithms operate on vectorized data. When applying machine learning algorithms to various applications that involve 3D geometry, it is critical to developing suitable 3D representations. This problem is fundamental as there are many possible shape representations, e.g., the triangle mesh, point cloud, implicit surface, parametric surface, part-based models, and light-field representations. Instead of focusing on a single representation, I study hybrid 3D representations that combine the strength of different 3D representations. Key advantages include powerful feature extraction and unsupervised learning of neural networks.

Our early results have revealed the huge potential of hybrid 3D representations. [CVPR19] and [TOG20] show that utilizing hybrid representations can boost the performance of 3D semantic segmentation and 3D scene synthesis, respectively. [CVPR20a] and [CVPR20b] utilized hybrid representations for pose regression and relative pose regression, respectively. In particular, [CVPR20a] improved the ADD(~S) score on Occlusion-Limond by more than 60%.

In [ECCV2020], we introduced H3DNet, an approach for detecing 3D objects in 3D scenes. The basic idea is to fit 3D bounding boxes to an over-complete and hybrid set geometric primitives, including box center, box face centers, and box edge centers. The approach achieved state-of-the-art results on ScanNet and SUNRGBD benchmark datasets. In [ICCV21c], we introduced a primitive shape detection approach that combines semantic features, geometric features, and edge features, for segmenting a point cloud of a CAD model into primitive shapes. In [ICCV21b], we introduced a new scene synthesis approach that combines the synthesis of object attributes and object pair attributes. In [ICCV21a], we introduce a new approach for learning deformable shape generators by combing a data term and a novel deformation regularization loss. Our [ARXIV22] paper combines an implicit decoder and a point cloud encoder to learn point-based features for self-supervised learning of point clouds.

On a theoretic side, we study principled self-supervision constraints for unsupervised learning. In [CVPR19], we introduced path-invariant bases for joint learning a collection of neural networks that forms a directed graph. In [NeurIPS19], we introduced cycle-consistency bases among an undirected graph and an algorithm for optimizing cycle-consistency bases via a condition number of the training loss.