PUGS: Zero-shot Physical Understanding with Gaussian Splatting

ICRA 2025
Yinghao Shuai1,8, Ran Yu2, Yuantao Chen3,8, Zijian Jiang1,8, Xiaowei Song1,8, Nan Wang1,8, Jv Zheng6,8, Jianzhu Ma6, Meng Yang4,5, Zhicheng Wang1, Wenbo Ding2, Hao Zhao6,7,8
1Tongji University, 2Shenzhen International Graduate School, Tsinghua University, 3The Chinese University of Hong Kong, 4MGI Tech, 5Chulalongkorn University, 6Institute for AI Industry Research(AIR), Tsinghua University, 7Beijing Academy of Artificial Intelligence(BAAI), 8Lightwheel AI
Paper arXiv Code Data
WeChat
WeChat QR Code

PUGS is a framework based on 3DGS which can perform dense prediction of physical properties for objects in a zero-shot manner.

Abstract

Current robotic systems can understand the categories and poses of objects well. But understanding physical properties like mass, friction, and hardness, in the wild, remains challenging. We propose a new method that reconstructs 3D objects using the Gaussian splatting representation and predicts various physical properties in a zero-shot manner. We propose two techniques during the reconstruction phase: a geometry-aware regularization loss function to improve the shape quality and a region-aware feature contrastive loss function to promote region affinity. Two other new techniques are designed during inference: a feature-based property propagation module and a volume integration module tailored for the Gaussian representation. Our framework is named as zero-shot physical understanding with Gaussian splatting, or PUGS. PUGS achieves new state-of-the-art results on the standard benchmark of ABO-500 mass prediction. We provide extensive quantitative ablations and qualitative visualization to demonstrate the mechanism of our designs. We show the proposed methodology can help address challenging real-world grasping tasks.

Framework

Overview of the whole pipeline. It can be divided into four parts: shape aware 3dgs reconstruction, VLM based physical property prediction, feature-based property propagation, and gaussianvolume integration.

Explanation of different modules in PUGS. During the reconstruction process, we compute the (a) geometry-aware regularization loss using normals obtained through two different methods. With (b) region-aware feature contrastive loss, we pull the features corresponding to Gaussians belonging to the same mask, while pushing apart the features corresponding to different masks. During (c) feature based property propagation, we use the similarity of region-aware feature to propagate physical properties.

Experiment Result

Mass Estimation Result in ABO-500

Ablation Result of GARL

GARL: Geometry-aware regularization loss.

Some Qualitative Result

Application about Grasping

BibTeX

@misc{shuai2025pugszeroshotphysicalunderstanding,
    title={PUGS: Zero-shot Physical Understanding with Gaussian Splatting}, 
    author={Yinghao Shuai and Ran Yu and Yuantao Chen and Zijian Jiang and Xiaowei Song and Nan Wang and Jv Zheng and Jianzhu Ma and Meng Yang and Zhicheng Wang and Wenbo Ding and Hao Zhao},
    year={2025},
    eprint={2502.12231},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2502.12231}, 
}