Contextual Outpainting with Object-level Contrastive Learning

1University of Science and Technology of China, 2Huawei Noah’s Ark Lab
Contextual Outpainting

Contextual outpainting: given foreground contents, synthesizes coherent background contents.

Intro Video


We study the problem of contextual outpainting, which aims to hallucinate the missing background contents based on the remaining foreground contents.

Existing image outpainting methods focus on completing object shapes or extending existing scenery textures, neglecting the semantically meaningful relationship between the missing and remaining contents.

To explore the semantic cues provided by the remaining foreground contents, we propose a novel ConTextual Outpainting GAN (CTO-GAN), leveraging the semantic layout as a bridge to synthesize coherent and diverse background contents. To model the contextual correlation between foreground and background contents, we incorporate an object-level contrastive loss to regularize the learning of cross-modal representations of foreground contents and the corresponding background semantic layout, facilitating accurate semantic reasoning. Furthermore, we improve the realism of the generated background contents via detecting generated context in adversarial training.

Extensive experiments demonstrate that the proposed method achieves superior performance compared with existing solutions on the challenging COCO-stuff dataset.

Demo Video


      author    = {Li, Jiacheng and Chen, Chang and Xiong, Zhiwei},
      title     = {Contextual Outpainting With Object-Level Contrastive Learning},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      month     = {June},
      year      = {2022},
      pages     = {11451-11460}


We would like to thank Yajing Liu, Shunxin Xu, Jialun Peng, and Zeyu Xiao for paper revision and Rui Li for inspiring discussions.